so hi welcome to our talk I'll introduce myself first my name is Nick Saunders I am our I'm
responsible for cybersecurity for Viasat government group I am also you know an engineer I've worked
in engineering for 15 years I've done I started in embedded systems I recently got to launch a
bootloader that I wrote into space which I was pretty proud of you know more recently I've also
kind of worked into you know upper layers of the stack and wear a lot of different hats in this
context though talking about these events I'll be sharing my role as one of the primary investigators
for the KASAT attack and so we will be going through these different series of attacks and
so Mark my colleague as well will be going through and giving us an introduction on these and then
I'll be taking it over for some technical
investigation and analysis and really telling our story for what this looks like
uh hi everyone I'm Mark Colaluca I am a network and security engineer by trade I did a lot of work on
layer two packet processing on some of the design of our first and second generation satellites
uh now I am the CISO at Viasat which in this context meant one of my great jobs was dealing
with the outside world to kind of give a protective bubble for all of our engineers and forensic
responders time and space to do what they needed to do so I'm going to go pretty quick through an
overview of the satellite network itself just to baseline everyone I apologize if this is a level
of knowledge many of you are familiar with but it's good to set the context so when we get in
the detail of the two different attack variants you understand the parts of the network so with that
how do I think
it's not moving
okay sorry
is that better sorry sorry guys
it's not our top and that's our tie yeah it's not moving at all
we may need IT help we made we made arrow buttons the space bar and the
yeah
there we go hey thank you yeah
if you're interested it's good to kind of Darren for like cicat�이 to the
So what are we talking about? February 24th, 2022. This is a graph of our terminals online
on a particular section of the KASAT network. The way the network is segmented, actually,
we have kind of by geography, but also to load balance, we have different segments of the network.
And you can see up until about 0300 or so UTC, about 60,000 modems online. And the two arrows
there represent the two different attack vectors that we're going to talk about today. The first
one, you can see a gradual decline in the number of modems. This ended up being a result of some
network-based attacks that we're going to get into in detail. They commenced there, but they
went for several weeks intermittently in different fashions, and you'll see how the attack actually
morphed.
And that ended up being a DHCP-based attack that manipulated both the protocol and the packets
and some other things. The second, which is probably the more widely publicized and discussed,
is the modem wiper event. So we're going to talk about the reverse engineering of the malware
strain and the toolkit that came with it, and kind of a little bit of the lateral movement
that led to the deployment of that. But that resulted in somewhere in the 40,000, 45,000,
50,000 modems being affected. There was a bit of a long tail on this as modems came back on the
line. We had some instability in the network because of the activity as well.
This is what the network is. Each of those are spot beams. Those are where the users live. So you can
see it covers most of eastern and western Europe and some of the Middle East. The satellite itself
was launched in 2010. It was used for both broadband and satellite TV access. And at the time
of the events, there was about 110,000 to 120,000 commercial modems on the network. And so
we now have
the customer base that we have if you notice from that graph there were still 20 000 customers in
the affected partitions still online still using the service the other partition had you know
upwards of 40 or 50 000 at any one time so we had a pretty big focus throughout the incident
response in the forensic analysis on preserving the stability and availability of the service
and that's a guiding point you'll see nick highlight throughout
out and with that i'm going to talk turn it over to nick to talk a little bit about the
background and some of the things that we were seeing early days and then he'll walk you through
both attacks all right so first word of warning don't believe everything you read on the internet
we're going to go through a couple of things that you know as we went through this attack
uh we were also reading the news as it was going out there and there was you know for good reason
in some cases we were not sharing information because there was
information that was you know close we were our main priority as mark mentioned was keeping users
online and uh you know ultimately patching any any findings that were out there but one of the
things that was kind of going on simultaneously was a few different pieces of reporting that we're
also here to want to get information out correct the record share get you know ultimately get uh
the details around what this attack was and was not so a couple things you know one was that the
attack was so effective that every one of the ksac ground user terminals that was turned on at the
time shut itself off are not the same as a lot of the other performances but we're going to look
off and could not be powered up that is not accurate we'll go through why that's
not accurate today you know also attackers were able to enter a
ground-based satellite network in which via SATs K sat certain users were
running by exploiting a vulnerability in the Fortinet VPN there was no evidence
that we found or that any partner is found and forensic evidence directs us
elsewhere it was not a zero data any knowledge or a vulnerability that was
known on the FortiGate there was in fact a FortiGate VPN we'll go through
that but that was not a vector and then the the third one malware was you know
part of a larger supply chain attack maybe if you defy supply chain
extremely extremely liberally then sure anything could be a supply chain attack
but there was no evidence of any compromise or tampering with a via SAT
modem software firmware images no evidence of any supply chain
interference for supply chain to via SAT so I want to correct that and then the
last one is the supply chain interference for supply chain to via SAT so I want to correct that and then the last one is
that you know this is my favorite one they deployed wiper malware to erase the
hard drives of the modems disconnecting them from the K sat network modems do
not have hard drives for the record I actually kind of like this one because
this is you know to be fair this is the more accurate one of these four but
modems do not have hard drives within them for the record the on a more kind
of personal basis my my experience as a as an engineer on this an investigator
on this was I got a call from my boss at 6 45 a.m. so thanks a lot
you know who here like 7 a.m. daily stand-ups can you raise your hand yeah
yeah was that your idea we had a bit of European partner and time zone issues so
unfortunately I couldn't accommodate everybody's request for a noon Pacific
so maybe next time it'll be different I don't know if I publicly say him but you
know so as as our teams you know a number of different cyber security people
got ripped
into that call because it was an apparent it appeared to be a cyber
attack so you know everybody was really eager to help let's get involved let's
go dive in figure out what was going on and what we found out very quickly on a
practical basis was you know Mark mentioned the case at network you know
that was operated by a partner that was operating that network so on a practical
basis we as engineers and responders didn't even have access you know we had
to log into we had to get access to log into that network and that network was
actually through that partner operated as we were later finding out you know
operating other networks as well through that same the same data center and so
there was some hurdles that had to be worked through and our partner
graciously did accommodate you know bringing us in as a extended part of the
team to investigate and so you know here's our hopefully chart which is you
know this is our wish list that we wrote up on the whiteboard of like a
couple other things you know later but you know the initial ones we just
wanted to get access to a jump box get us into the network you know get us
twenty
port 22 access to terminals so that we can investigate so we talking through you know we
wanted to be able to connect to a terminal over the air log into it and diagnose what what uh what
was going on for some of the terminals that were online if i can add there too i was very unpopular
with the engineers because one of the first actions we did was hit the giant red button which
only not only segregated severed all of our management access to kasat but also to other
via sat networks around the world because we weren't sure if other ones were vulnerable or
at risk or under attack so i was extremely unpopular with a large amount of the ops and
the forensics teams because i made them crawl through broken glass to get access no comment
so i'm going to take you through our knowledge what happened initially so what did we know
what did we not know so mark shared that graph earlier that was one of our initial pieces of
knowledge the next thing that we learned was that there were complaints from distributors
that some terminals would only light up with a
solid white led which is not normal and then the second piece of knowledge was conflicting which
is that other terminals would light up with a blinking blue led which was another abnormal case
but that they do not provide communications or enter the network and so in both of those cases
those were not normal conditions i didn't know what these lights meant and so you know first
rtfm like let's take a look at the instruction manual and figure out what does it actually mean
when you get a blinking blue light or a like a solid white light and so if you look on here
solid white means initial power up that's an early stage that's not good you know the other
case that we reviewed was the the pulsing blue case the device is busy but working normally so
that means it's booting up i guess but you know it's busy so i didn't really know more detail than
that to begin with and so i'm going to step back a little bit before we go into a ton more because
one of the things that we had to do that i had to do that other incident responders had to do was
we had to come up to speed with a legacy network architecture we've got multiple different
generations of different network architectures and so we had to come up to speed so i'm going
to bring you up to speed i knew a lot of this but you know we all had to level set and so we're
going to level set in this room together on a satellite communications network kind of this is
a little bit biased toward the way we do it but in general we're trying to stay a little bit you
know provider agnostic in this view so typically excuse me typically you have you know customer
premise equipment um that's like an end user device that connects over hard wire or wi-fi to
a terminal or a modem that interfaces usually with a home through a router that's also coupled
with that modem then you have an antenna that communicates over rf to a satellite i like to
think about a satellite as a big mirror i mean it basically does frequency translation from one
frequency to another and then you know beams that back down over a different frequency band
but really it's the same signal and then it goes back down to a ground gateway system which then
is responsible for converting rf into a mac layer
and then you end up with a core node that's often providing depending on the network maybe layer
two layer three services and then you have different sophistication levels of a you know
of your core node as we call it um that include different things and so going through those
different elements you have a control plane which is you know for us a control plane really means
signaling to a device you know you know communicating with the device
getting the network to function uh making changes that's controlling of
a network so that might be instructions down to a terminal that might be communication of a terminal
to authenticate itself with the network and that's where you see our first bullet
triple a so every single modem on our network on the kasat network in this case but on every
network that we're running has an identity that identity you know terminals have their own private
key they have their own public key they authenticate with the network through a triple a
server that is responsible for making sure that when entering the network a terminal is authentic
that you also have dhcp dhcp as we know is a protocol that's used for giving an ip address
for example but many other things as well if needed so you can do configuration of different
settings and so there's a lot of different parameters around dhcp that is actually one of
the areas we'll we'll be diving into throughout this uh throughout this presentation and then on
the management plane side you have other things so you've got device management that might be things
like pushing software images down to terminals through distribution mechanisms through image
and things like that that also includes you know as i mentioned earlier getting ssh access to
terminals we were able to use ssh to servers to terminals things like that to interface down with
different systems within this network and then server router administration typical things to
be able to manage the different systems that are involved in that corner and ground network
and so that's that's kind of what we call those is our ground network is kind of the the summary
of that little element and then those connect a user to the internet or
in some cases maybe private networking setups and so um that's that's an overall topology
i will pivot toward the kasat more specific network layout and so kind of knowing all that
you know now what that what a data plane what a control plane are what a management plane is
this network was made out of you know if you look at the left-hand side of this diagram to
orient you again are the users there's users with modems or terminals and those users in
the kasat example were served by different partitions of a network and so we call
those bandwidth aggregation points um in the kasat context those are um bap1 bandwidth
aggregation point one served a regional set of users uh shown on the left those are commercial
users bap2 was also a different set of users um and then bap the the last one was a viasat
operated mobility core which was also it included you know different systems and one of the elements
on this was there's a partner operation of that bap1 and then bap2 and then we were responsible for
that and there's pretty much a different core node network that was assigned to that
management network of the viasat operated one shown down on the bottom for mobility and
government cases so how many people in this room know which side of the network the attack came
from so i'm going to ask audience participation raise your hand if you think that the attack
came in from the left-hand side of this diagram okay raise your hand
if you think that it was on the right-hand side of this diagram okay everybody's wrong because i
didn't see the same people raise their hand from the second maybe maybe one person okay
so i'm going to go through first everybody's familiar
i think generally with the reporting around the wiper attack so we're going to call that
method number one the wiper attack that's been very widely reported so we'll go through this
we're going to go through our investigation of this but the other side of this
was a different view of crafted attacks that came from terminals we haven't gone through those
widely and so we're going to break those apart and what those crafted attacks looked like which
were focused around denial of service activity on the network so we didn't have i'm going to be
fully honest that oh it's on this side okay
okay we'll deal with it
sorry guys the right side of the room is now in the wrong um okay so i'm going to be fully honest
this this team structure developed over time as we learned so as i mentioned we only knew a couple
things at the very beginning we knew the lights weren't turning on correctly right so we had to
create these teams dynamically on the fly but i'm gonna i'm gonna talk through these
different efforts because there were a lot of subject matter
and so we came up with a number of different sections to do this this was our first step
we had to provide a lot of great experts engineers involved that all deserve a ton
of credit for the work that they did and so the analysis that we're going to go through i'm going
to represent the work of a lot of others that were investigating on these different investigation
efforts and so that was our kind of typical like high-level team layout for these different
response efforts so we're going to dive in to the wiper attack investigation first
and into our Carlsbad Labs. That was in California. We wanted to bring them in, ship them in from Europe, get them in.
And so we shipped them. We had to get them through customs. Friday, we were supposed to get them.
Customs attempt number one failed. Somebody got turned away through customs, and honestly, come back tomorrow was, I think, the answer that happened.
And so not knowing all the details, our first attempt to get these terminals into country failed.
And so we got them on Saturday and laid them out all on a conference table in a room.
So what we did on this conference table was there were different sticky notes.
There were different pieces of information kind of annotating these different terminals.
You, Mark, had requested kind of, do you want to go through kind of what you requested from the distributors?
So since we didn't really understand what was resident on them or what it talked to them, we have similar type modems on other networks.
Also, this was.
Completely air gapped room, but we want to make sure from a protocol standpoint that we did not tamper with or adjust or inject anything that wasn't there when we received the modem.
So I required them to establish a protocol specifically, for example, of no transmission pins were being enabled when we did the reverse engineering or the analysis because we wanted to make sure that what we were getting as output was exactly what was there and we hadn't tampered with or altered it in any way.
Okay.
Cool.
So as these modems were arriving, one of the things we were also doing was what what's our plan of attack?
Like, what do we want to do when we first get these modems?
And we constructed a plan, wrote it up on a big Wiki page and then said, okay, here, when we get these terminals, here's the very first thing we're going to do.
Here's the second thing that we're going to do.
And so the first thing that we decided to do was we want to engage with these modems through console.
We just want to pull logs.
And so we had to do some hardware outfitting to change these terminals so that we could connect to them through console.
Shown on the video on the right is actually a modified version of a UART connection because of one key constraint that we developed in our plan.
We did not know any state about these terminals.
There was no wiper write-ups on the Internet.
Nobody knew anything about this.
So kind of putting yourself through our lens, we did not know if there was code on there that as soon as you connect a TX pin that might transmit and it might wake up and then adjust.
The way that it is behaving, wipe itself, you know, we did not know what was resident on these terminals that we got in.
So we wanted to be very careful not to lose any state because we all know how hard it was to get something through customs.
So we didn't want to mess that up.
So that's a three-pin UART where you might otherwise see four.
And so what we first did was we pulled logs, and this was one of the terminals where it was a white light terminal.
And so that white light terminal meant it was an early boot failure.
What we saw in our...
experience was that on the left-hand side, the not working version of a terminal actually did not spit out logs past a very early stage of the boot sequence.
And so what we did was, okay, that informs the way that we're going to think about these further.
Our next step in the process that we had identified was we're going to look at JTAG.
And so we're going to connect to these boards through JTAG.
So that's exactly what we did.
We got our hardware teams to outfit these boards properly.
There are a number of people that made some changes to properly get us...
a JTAG connection so that we could do a boundary scan, do a JTAG connection to these boards,
and to pull the chain of what was actually there on the...
that was all connected to the JTAG chain.
So we did that, connected.
There are a couple of different versions of terminals.
And so we'll talk about the different variants of those terminals later, but we had to do some different outfitting for kind of the different versions of those terminals.
But the approach that we took
was to modify U-boot to dump Flash.
We wanted to see what was in there.
We wanted to get everything out of that terminal because we knew that like the early stages,
you know, if there's a boot failure, the boot sequence would ultimately read from Flash.
We knew that we needed to pull Flash content.
So that was the first thing that we decided to do from there.
So after this, we started to pull content from Flash and we were able to see, okay, this is a little bit of a weird set of content.
The audio will be connected to this.
So let's see.
Okay, so I'm going to, there is a, on this side, this is the very first time where we compared the content of Flash from a non-working and a working modem.
And so what's shown on this is we're looking at the left-hand side computer that you'll see here.
On this screen is a working terminal.
You can see there's multiple different bootloaders and there's multiple different stages of bootloaders in a different file system.
So there's a bootloader file system.
That bootloader file system is a bootloader file system.
The bootloader file system included the contents.
And so that would, those are images that U-Boot would ultimately boot into for a stage two and a stage three boot sequence.
On the right-hand side, you can see the Flash content of the first stage bootloader.
And then you can see at the very end, you saw it before, but as we scroll through this, you can see a decrementing value of the Flash content.
And so this was our initial, this is the first time that we ever saw Flash content on this board.
So that's a, that had a couple of things to do with it.
A couple of unique things about that.
That was a decrementing value.
That's not just, you know, something that could have gone wrong.
There was a decrementing value that was the content through Flash.
So that looked wrong.
And then the last thing that you'll see as we scroll to the bottom here is there are also no contents in the file systems.
There's no partition table for either of the file systems that those should then read from and then boot into the next stage of the bootloader.
And so all of that looks wrong.
So that was our first readout.
And that happened.
We learned that on Sunday night.
So the attack happened on February 24th for this wiper attack.
And then on February 27th, Sunday night, we worked through Saturday to Sunday.
Thank, so many thank yous to every engineer that was involved over the weekend.
People were diving in to help.
But what we did was get a good understanding of the content of Flash.
One thing just to note that Nick just mentioned too is the removal of both file systems.
That's pretty typical of modems of that generation to have, like, last known.
Good.
So both of those file systems appeared to be dropped such that if the modem rebooted, thought it had a problem.
If it had the ability to get past first stage, it would go back to a known good image.
So it appeared that both of those might have been been removed or at least altered.
Well, Mark, do you want to take this one?
Yeah, I'll go really quick.
Yeah.
So during the weekend, while we're doing the hardware analysis on the modems toolkit was found on one of the management servers.
If you remember that management plane in that.
In the core node, the toolkit was a that's the actual name of the file, if you see it, and it was dropped to us on 3 42 a.m.
Sunday morning.
It was retrieved that central time, and that toolkit contained a number of scripts as well as some, I'll say, suspicious looking binary files.
And so we ended up kicking off one of those other threads that Nick mentioned, kind of the reverse engineering and analysis team.
So Nick's going to walk you through what we did with the tool.
Kit, which includes figuring out the sequencing of the contents of the toolkit, reverse engineering, the binary itself, and then a detonation step to actually verify that it produced the same kind of outcome that you just saw on there.
And in parallel, we actually as part of our standard incident response mandates, our forensics and investigation partner, they convinced.
And on March leaf Monday February 28 is when we engage them, and I think the very next day is when we gave them.
And so we're going to talk about some of the things that we're going to talk about today, and then we're going to talk about some of the other things that we're going to talk about today, and then we're going to talk about some of the other contents of that as well, so they could start their investigation and forensic analysis.
Well, so, as I mentioned, there are a couple of key questions and and we didn't know a ton in the very beginning, so we had a number of different questions about the toolkit that was found and we had we had a lot of questions and a lot of those questions affected the way that we were thinking.
So some of those questions were what other capabilities did the toolkit have that have evasion capabilities was it selective over targets.
How did it erase flash could anything be done to stop it could walking dead modems be recovered could itself propagate.
There's a lot of different questions that we had about this toolkit so we wanted to understand what the what what was actually there, so this was.
We spun out our toolkit reverse engineering team, which consisted of gentlemen, I really do want to give credit for.
Credit to Jonathan wyatt and a couple of his colleagues Andrew marsh these these gentlemen were able to quickly use an NSA open source tool Deidre.
To investigate we'll go through the the investigation that they did in a second from that toolkit.
But one of the first things that they did before that was to do an emulation through came up, and so they installed the the.
One of the key binaries that was found in this toolkit i'll go.
Of what was contained in that toolkit on the next slide, but they emulated the way that this malware or this binary would actually behave.
Through came in initially, and so they were able to see a distinct pattern we recognize that pattern from before from what was actually recovered from the modems that were in the field.
And so, another key question that we had was was this malware actually the toolkit or was that a decoy we didn't know, so we had to pair those two things together.
So on this slide, this is a picture of what was in that toolkit from a summary perspective.
So, as you look at this toolkit there are a series of files and scripts.
All kind of centering in around use of SSH and SSH credentials to distribute an executable down to a terminal.
One of the key things that I want to highlight about what was in this toolkit was there were a couple of different ways.
That these attackers were you know interacting with their scripts there is a series of files, some of them had gotten changed initially it seemed like and we're speculating.
Initially, it seemed like there was a hardware list we don't know the order in which these scripts were were executed, but there was a hardware list.
On this network based on what everybody in this room now has seen there's a satellite communications overview.
This is not one layer to broadcast domain so that you know doing an art based lookup for terminals doesn't actually even work.
On this network so looked like that attempt was then you know changed into it in favor of a different attempt and so there's a series of different scripts and files that were all centered around distribution of this executable down to a terminal and so we wanted to figure out what was actually in that executable in that.
Well, so we want to figure out what was actually in that executable and so, as I mentioned, we use the NSA tool Ghidra to understand what was actually in the executable what did it do.
How did it work does it make the same pattern and so one of the things that we've quickly did was it takes if anybody's used the tool before you have to spend a lot of time renaming functions, you know understanding sys calls understanding numbers and so this is kind of the derived version of sys calls mapped back up into an understanding this was created through an internal document by the internal document was finished on March 2 within our team so really proud of that team for their work.
And so what we did was we also let's detonate it let's let's let's take let's get some sacrificial terminals and so one of the things that's really cool about this.
This event was the way that the teams rallied to be able to respond, we were able to quickly acquire terminals from a lab that we had on a shelf and detonate so sorry terminals you guys are you're the unlucky ones.
Some of these terminals were then pulled into our lab our response room and we detonated we distributed the the kit down.
Executable down onto those terminals, it was a mixed binary we executed that mixed binary on the terminals and we're able to see that it does actually have the same effect as what so so now we had three different independent signals that all correlated together to understand that this toolkit was in fact the toolkit that really did help us to understand what was actually there what what what everything was.
So just to kind of summarize our initial.
Attack conclusions we went through and developed an understanding of you know the attacks happened at three to 4am utc i'll go into the other attack which happened around 3am utc on Thursday to 24.
Modems arrived on Saturday Sunday night we had discovered flash content we had read that flash content understood it.
And then our teams, then we're also able to reverse engineer that the toolkit and understand it over the series of days they were kind of streaming and information on that.
That reverse engineering.
Process because oh cool, this is what I found that okay it's it's really looking similar to that and that internal final publication of that document or that final document was was kind of created by the the March 2nd time frame.
Mark.
Kind of put the conclusions to a picture.
For us.
Based on the forensic analysis and a little bit that came afterwards, but essentially attacker through known tour node.
Exit relays.
Unauthorized access to that vpn with compromised credentials pivoted to management server which actually a different set of compromised credentials.
Went to a network ops server to do some recon during this time period that's the stuff about trying arp and hey what at what modems are alive or can I reach which parts of the network.
And then pivoting to the place where we typically do distribute software images but just using that particular place to.
Stage the toolkit and.
This is it kind of pictorially from a timing perspective what we saw actually.
The first attempts were about five or six o'clock in the evening trying to get access to the vpn at several failed logins.
First successful compromise with another unauthorized access about eight o'clock in the evening.
Then took about two hours I don't know if the attacker attackers needed dinner put kids to bed whatever.
Uh.
Went to a management server hung out there for a little bit.
Then went ahead and said I'm going to check out the network right around ten or eleven o'clock that evening and then about eleven thirty through one a m two two a m went to that.
That server started staging the toolkit and then started executing attempts but it looked like about four fifteen as you saw the era with the precipitous drop is when.
Essentially sent out this over the satellite network just.
Going modem to modem.
down the row and sending the toolkit placing it on the modem and then executing it once the binary
was executed it performed the flash memory overwrite that you saw with that pattern and
then attempted to reboot the terminal which would put it in an inoperable state okay now we're going
to talk about the other the control plane attacks if you remember these started at roughly oh 300
utc and i want to emphasize while they were going simultaneously throughout that morning
there was some intermittent behavior and they also continued for several weeks as some of the
methods and means that were being used we had mitigations deployed they would pivot to a new
technique so i think we've seen this slide already just to orient now we're talking about the second
method um and so this this sec the second method was crafted attacks that came out of terminal so
we're going to decompose that and what were those crafted attacks and what when we learn from those
uh we have since gotten the
chance to apply a lot of all of these fixes onto our networks and then i've also been working to
generalize the understanding to say okay what other attacks could look like this and so that's
what we want to get out there and inform the community of i know i'll add to that this is
one of the places through private sharing forums and direct communications with other satcom and
service provider this this is a more i'll say a more similar pattern that other service providers
might use to provide control plane or signaling or ip addresses so
these are specifically manipulating things we use in our network certain dhcp option fields etc
we thought it's generalized enough that we should share so this was some of our earliest sharing
both directly with satcom operators as well as through our government sharing partner one thing
also to call out on the prior slide is that this fell then into the other track that we had formed
as part of our incident response which was the network and infrastructure analysis and response
teams and so those teams were a series of subject matter experts and security
engineers that were all kind of responding ultimately to these network based attacks and
so this was a distinct track from the others but we were generalizing the learnings and
understanding and aggregating them into a composite picture the whole time
so as we go through this we're going to look at this first attack example so in this first
attack example we've attached a timeline to initial dhcp request volumetric attacks that we
saw starting at 302 am utc on february 24th if you recall the prior graph that mark mentioned
you saw that initial drop off and then you saw a steep drop off this was around the same time
of that initial drop off of terminals when you started to see that was due to effects
to the network entry process that were caused by these crafted attacks coming from terminals and
two other points i just want to make on this one one if you see the ledger
in the top right i mean these are legitimate modems with valid subscriptions all throughout
that beam pattern that there was no localization like they were all in one place or on these are
just essentially random modems that appeared to be doing this the second thing to note on there's
that gap in the middle it looks like it kind of stopped well part of that was due to the
instability in our network because of the volumetric attacks some of the elements that
nick's going to describe stopped responding crashed reboot analyzed take it offline take
a note and then we're going to go through the rest of the attack and then we're going to go through
node out of service
cool so now i'm going to dive in i'm going to decompose a control plane a little bit more
than we have so far so you have a customer premise equipment you have a device of an end
user you have a terminal then you have a gateway so that terminal communicates ultimately through
the satellite through the gateway and then that gateway makes a decision depending on whether that
traffic is control plane or data plane and then that that traffic ultimately flows through
the network and then that's where the network goes so we need to do a lot of work on that
so what we're going to do is to we're going to go through an asn is a role within a system
architecture for the network this is a virtualized asn that then that system then acts
in the dhcp context as a dhcp relay that's an extension to the dhcp rfc and then there's dhcp
servers that are facilitating dhcp requests and responses there's a aaa as i mentioned for
authentication and servicing of that and then there's a couple other systems that are involved so
So as we decompose this, this is a little bit small.
It's an eye chart, but I'm going to talk you through this.
And so this is attack number 2.1 of three different attacks that I'm going to decompose
that we saw happen on our network that were these crafted attacks from terminals.
So the precondition for this attack first is that a terminal must have valid access
and an identity within the network.
So you must have a terminal-specific private key.
You have to be authenticated within the service.
You're real.
Yeah, so terminals that might have been stolen or haven't been provisioned yet,
they don't yet have that kind of key, and they go to a walled garden for an installer to go install it.
So you can't get to this place in the network, if you will, unless you're a terminal with a valid subscription.
Prior to that, if you hadn't paid your bill or the service was suspended,
AAA would send you somewhere else.
So this is a pretty important point.
This has authenticated itself to the network using its key,
and it's got a valid subscription.
So from that precondition now, you see the next, you have a user device which issues a DHCP request up to a terminal,
and then that terminal then sends that DHCP request on.
This is a terminal that in this case is not acting as a router.
There are sometimes cases where if you've got a router involved, then the router might act as a relay for that.
So this is the case only where it's assigning an IP address.
Essentially, think of your home router, basically.
The home router gets an ISP.
The IP address, that is that request, right?
So that DHCP request is then filled out.
It's sent across to the gateway.
The gateway then propagates that to a VASN,
and then that ultimately then gets relayed to a DHCP server.
So in that DHCP request, in the DHCPRC, I think 2131,
it defines a client address as really like it's the CH adder field is essentially an opaque key.
That opaque, excuse me.
That opaque key can be anything that you want.
It's essentially a hardware address in this context for a modem.
That then might include, there's kind of two variants to this attack if you want to execute it.
One would be to set your client hardware address to any,
and then associate a requested IP address, which is DHCP option 50,
with the IP address of a target terminal.
So this is what we saw from the attacker where there is IP addresses of target terminals,
that were in these volumetric attacks.
You could also, when we reviewed this, you could also do the reverse.
And so when we have since then applied that into a patch,
we've had to think about and generalize what are the other cases that we didn't see,
but could also then be used.
And then kind of winding ahead to the DHCP server,
you can see a DHCP request to a DHCP server that says the request is invalid.
It doesn't match an active lease.
And then a NAC is then sent from the DHCP server to the VASN.
That VASN system then is responsive to that NAC.
It shouldn't have been in this case.
That was the problem.
So the VASN takes an unsolicited NAC,
and then is confused into issuing a command on that control plane
to disconnect a terminal from the network,
and that ultimately would knock a terminal offline.
So it was a fairly sophisticated way to use multiple different,
essentially DHCP system roles,
to confuse multiple systems.
Or systems operating together.
And so if you think about this within buzzword zero trust context,
you have to think about what are the ways that you might be able to confuse systems
where the DHCP server is implicitly trusted from a VASN.
So you have to think about how might that affect something else,
and how can you ultimately prevent that from happening.
So I can wind pretty quickly through the rest of these.
You have a second variant of this attack,
which is a DHCP decline attack, which we saw.
Which is ultimately issuing a DHCP decline for a target terminal.
That then causes a DHCP server to really not care about that.
That actually confuses a VASN to then disconnect the terminal.
We responded to this and then were able to disable this request.
And then lastly, same exact attack,
but through a DHCP release method as well.
So this is a bit of a cat and mouse game.
From a response standpoint, we saw one attack.
We saw a DHCP request attack, we saw a DHCP decline attack,
and then we saw a DHCP release attack.
Every time we would make a change, we would then see within days response,
and then within hours as we got better at it.
And then the responses would change over time.
And then later we applied this into the threat modeling context,
and then ultimately hardening of these different systems.
I will say just the attacker or attackers
also displayed similar responsiveness time
when they definitely had ways of testing and viewing and observing
what changes or behaviors that we made,
and they responded pretty quickly.
So what specifically did we do to defend?
Wouldn't an adversary like to know?
But the things I will share here is that we quickly applied filters
to control plane systems to intervene, disable these requests.
We set up a dedicated control plane slice.
We had engineers dedicated to this,
to set up a dedicated slice that was a Ukraine-only control plane slice
to keep that segmented from the rest of the networks on the fly.
And so the control planes were then separated at that point.
That was a pretty prideful moment when we had that new control plane slice online
and dynamically responding to those addresses.
And then lastly, I'm going to just kind of give a shout out.
There have been a lot of different engineering teams
that have been involved in RF attacks.
This is not...
So we've seen attacks from different cases of attackers
that are out there around RF attacks as well.
It's not just these domains that we've had to worry about.
And so this has been a very ongoing series of events for us.
We've seen eight different changes of RF interference over the last year.
So this has been a pretty active domain.
So our ask...
I was going to say, no, I'm ready to...
Go ahead.
So our ask...
You know, service provider networks, especially SATCOM,
are especially complex.
You know, LEO, GEO, whatever that is, right?
They're complex systems.
So as you think about those,
you know, we know what complexity introduces in the community, right?
That introduces vulnerabilities.
And so you have to think about the system complexities.
And so one of the things that we want to share with this community is our ask.
So our final kind of slide, our ask.
We have a traditional bug bounty program through BugCrowd.
We typically have campaigns that are kind of user-facing things,
like billing systems or customer support portals.
In some cases, it's our corporate infrastructure.
So we're announcing an expansion for the research community,
kind of if we get vetted researchers through BugCrowd.
We're going to ask for some help by providing terminal kits in service to selected researchers
to be able to test over-the-air methods and attacks
for disrupting a particular canary terminal in there.
This is the kind of thing that we would love to engage the research community,
to think of ways that maybe we haven't considered yet
to manipulate or interact with a terminal.
So basically, interest in resume portfolio for our Responsible Disclosure Program.
We'll take submissions in September, kind of review those,
and select some researchers, hopefully, to have some fun with our network.
So I want to thank everybody for their time and attention today.
I appreciate it.
Thank you.