Before I get going, I really want to appreciate Queenie and her crew.
Let's give them a good hand for the job they are doing.
It's hard work. I know they love it, but it really makes a big difference for us.
So, thank you guys.
Okay, running system tests with active authentication authorization.
That's the theme of this talk.
So, this story starts on a Monday.
I'm just into my second coffee cup, getting ready to start the kind of new week.
And I got a new email.
Someone has been crawling around our website, hitting the well-known endpoint and the security XT.
There is a security researcher that kind of asked me,
what tells us that, hey, I've discovered this kind of stuff.
Do you mind taking a look at it?
So, during the post-mortems with the teams responsible for the services, we discovered a few things.
This was a test system that was kind of open to the internet.
It has a lot of the kind of security features turned off.
And it contained a bit too much data, typically data from production.
So, like we do in our AppSec team, we start working together.
Together with the teams, it seems to be a recurring issue.
And together with the teams, we try to define a few patterns and examples and so on,
how we can, yeah, moving this kind of issue forward and making their life easier.
So, this is one of the stories I'm going to tell you now, how we are working with these things.
So, the objectives for our teams is to encourage them to divert more effort onto system testing
with authentication and authorization.
And all the other security features activated.
Activated, to me, means like turned on, like they are in production.
And those of you who are programmers, like me, you know how hard that really can be.
It's not an easy task.
So, why do we pursue this kind of objective?
It will have, and it has shown us, to have an impact on mitigating well-known security risks.
I'll just kind of show a few ones of the OWASP Top 10 and the API Top 10.
You know them.
But these risks are in our systems.
And with the patterns that we are showing, we hope that some of our teams will mitigate those risks.
So, a disclaimer before I continue.
The more I learn about the topic at hand, the more I really, really realize how complex it is.
I am not an expert.
Life is not simple.
All solutions have trade-offs.
And there are always, always more than one solution.
And context matters a lot.
You can't just say, like, do this and do that to any kind of teams.
You have to take into consideration the context where they kind of live.
After this, I will show a few code examples.
You will get access to the slides and the code and everything.
But please, this code is not production quality.
I've used some part of the code to do some training on potentially bad patterns.
So, don't put this stuff into production if you kind of decide to take a look at the code.
Some of the patterns that we discuss here are kind of risky.
So, be cautious about it.
And what I really recommend to our teams and to you guys, whenever you have, like, a test system and the whole software development lifecycle, please do a threat model on it.
Ask yourself what could potentially go wrong and so on for the SLC as well, not only the end product.
Who am I?
I'm Lars, an application security advocate at Equinor.
That's me.
It's a Norwegian company.
I've been around for a while.
I have a smashing wife and three grown-up kids.
And, yeah, you get all the details here for where you can connect with LinkedIn, Twitter, and GitHub.
I think Twitter may be the best place, or GitHub.
Outline, what are we going to look at?
Some observations.
Then we're going to take a look at the demo application.
Super advanced system.
We're going to discuss things that I've done.
It's called test principles and technical challenges.
And we're going to look into exploring one potential solution to solve some challenges, wrap it up, and have a Q&A at the end.
And slides and code and all that kind of stuff will be available for you guys.
You can play around with it as much as you want to.
I like this kind of quote in the end.
It's not mine.
I've probably stolen it somewhere.
There is no end state for application security.
We just learn and improve.
And that's what I hope you guys get out of this session as well.
Maybe not everything is fit for your kind of context.
And maybe there are a few points that you can take away and bring back home.
And not necessarily Monday, but whenever you're back home, making the life for your developers a bit more easy.
Helping them become more secure.
So, some observations that we have made from our postmortems and discussing with the DevOps developer teams and so on is as following.
Our team reports that 80% of our test effort is spent on typically unit testing.
We do very little of system testing.
We also see that authentication and authorization are either mocked, stubbed, or disabled when testing.
That's not an unusual situation for many of you, I think.
We don't write a lot of security-related tests.
We are, to a large degree, focusing on the happy path and the functional tests.
In testing, security features are often disabled or mocked, as many of you know.
And in modern systems, quotes, especially when we are hurrying into the cloud, which I guess most of us are.
And security is shifting left, meaning that the development teams are getting more responsibility for the security.
Both of the infrastructure, for the pipelines, and for the end product, and so on.
We...
I think...
At least in my opinion, should do a lot more system testing.
Because we want to test everything as a system from the customer's perspective.
But it's really, really hard.
This challenge has multiple kind of perspectives.
Not only the technical one, which I'm going to focus on now.
I really acknowledge that things like personal skills and proficiency, the team compositions, the team capacity.
And not the least, the culture in your team.
And your company needs to be taken into consideration when you decide what to do.
They are super important.
We can't only do the technical stuff.
There's a lot more.
My context and my culture is primarily taken from the Nordics in Europe.
So, some of the stuff that I say may not make sense to you.
And maybe that's one of the explanations.
So...
Yep.
The demo system that we're going to look at is something like...
This.
It's a super modern application.
It's kind of basically giving you a Game of Thrones episode.
So for the first season, you get a list of episodes.
And you also get an awesome quote from Game of Thrones at the end.
And you can read your mailbox from Outlook.
The architecture looks something like this.
How does this work?
Not too bad.
Something like this.
So what happens is that we have a backend.
That's a client here.
Which is kind of serving the frontend to the client.
That's a web thing.
The backend client has a session store.
Which in our example is in memory.
It has a token cache file to manage the access tokens that are gathered.
And we're using the frontend for backend pattern here.
So there are no tokens stored on the client and so on.
We have the episodes API.
It's like...
It's an API on its own.
And we also have the quotes API.
In our context, we use Azure AD as the identity provider.
So Azure AD handles the authentication of the users.
The access stuff and so on.
And we also read information from the Microsoft Graph.
That is typically the inbox and so on.
When we do the authentication and authorization.
We use OpenID Connect.
To do that stuff.
And we are using the Microsoft authentication library.
On the client backend to facilitate those things.
It's a backend for frontend pattern.
Which I mentioned.
And the basic flow is so you can just get your heads around it.
Or try to visualize it.
Is that when I hit the login in my client.
It will hit an endpoint back at the backend.
That will redirect to the browser.
To Azure AD.
Hand over the authentication to Azure AD.
If everything is okay.
Access tokens are sent back to the client.
And the stuff that is in the web browser.
Is a token called session ID.
That's the connection to all the data.
And of course if I have the session ID.
I can use that one to retrieve the various tokens.
For the login user.
So when the client wants to get the episodes.
What it basically do is to.
On behalf of the login user.
Contact Azure AD.
And ask hey can I get an access token.
To access the episodes API.
If that's kind of okay set up.
It will return an access token.
And the client will use that access token.
To hit the episodes API.
The episodes API will then verify the token.
To see that it's kind of valid and okay.
And all those kind of things.
And prepare to serve a list of episodes.
When that is done.
The episodes API.
Will on behalf of the login user.
Request another access token from Azure AD.
To request quotes.
And if all that stuff is kind of okay.
The token are validated okay.
And so on.
All that information is kind of sent back to the client.
And put into the front end.
So the reason why having a few of these technical details.
Is that this is a typical scenario.
That we meet in our settings.
We have like a client stuff.
But we touch APIs.
And chaining and so on.
So this is not unheard of.
That's why I call it a modern system.
The future that we're going to focus our testing on.
Is like using the browser.
To log in and retrieve.
A game of thrones episode.
For the login user.
In the GitHub repo that I'm sharing.
With this stuff.
There's a more detailed discussion on the scenario.
If you want even more details on that part.
But before we kind of go in.
And look to the solution.
Let's discuss a few test principles.
And technical challenges.
So test principles.
I typically advise teams.
To have a few of those themselves.
And we have selected a few for our scenario.
So yeah.
One principle is to be that we should automate.
As much of our testing as possible.
Non functional tests.
Are as important as the functional ones.
A third one.
Don't test in production.
Maybe that's a bad idea.
Even though someone is.
Saying that it's okay in some cases.
Use synthetic test data.
Is another principle.
And running code should not know.
That it's under test.
And testing should not.
Involve components.
That we don't control.
So this is not an exhaustive list.
It's just an example of some test principles.
That we have set for our example.
And.
You will see that maybe.
We have to break or challenge.
So what are the technical challenges.
With testing this.
From like the end user perspective.
Through the browser.
That is to do the login through the browser.
And that's a super challenge.
It has a lot of challenges on its own.
One of the typical one.
Is that the authentication.
Is not handled in our client.
Is handed over to Azure AD.
So now I'm leaving my own browser context.
And most bed drivers.
And browser will not.
Stop me that.
Because it's a different domain.
So that's a very typical challenge.
That we face.
And there are no tokens in the browser.
So we must kind of mess around.
Slash utilize the back end.
To get this testing going.
There are some examples of applications.
Where the whole application is stored.
In the client.
And the tokens are stored in the client.
Client being the web browser.
And then you can mess around with them there.
But that's not the pattern we have implemented.
Could we acquire tokens.
And manipulate the token cache.
The cache that MSAL is providing.
Yes that could be done.
But it's really really hard and messy work.
So I'm not sure I would say that.
Is a good way of doing it.
But it could be done.
It's super messy.
Could we mess around with the session objects in the back end.
Yes our test is driven from the back end.
So we have access to both these kind of things.
The token cache and the session cache.
And maybe there are some opportunities there.
And of course we have tried it out.
And again it's super messy.
And the corporate environment.
Which I come from.
Working in a Norwegian company called Equinor.
We have MFA enabled for all of our users.
And that makes.
Kind of a automation.
And automation of login.
Slash testing.
More or less impossible.
And of course.
If you only have one kind of identity provider.
For all your users.
Test users or not.
Then we are also using a production system for testing.
Which is a very good pattern.
I would say.
So yeah.
Just mentioned that one.
Could we mock the identity provider.
With an OI2 mock server.
So that's a solution that we have tried out.
And it's definitely doable.
But it depends on what framework.
We are using on the back end.
If you are using Microsoft MSAL framework.
And try to use.
Like an OI2 mock server.
To mock that one.
You are in for a real treat.
Because Microsoft's framework.
Is Microsoft's standard.
So the OI2 standard.
Plus the Microsoft stuff.
So you really have to mimic.
All what MSAL expects.
In the request responses.
In your mock server as well.
And before you know it.
You are more or less developing a parallel thing.
To MSAL.
It can be done.
But it's really really hard.
So the choices that we make.
Should as far as possible.
We have the security features.
Of our system.
That's one thing.
And we would like to develop.
As little as possible.
Of the testing support code.
It's not the test code itself.
But all the stuff that we mess around with.
Try to kind of enable the testing.
There is also a more rapid discussion.
Of that one available.
In the GitHub repo.
That I will share a bit later.
So time to make some decisions.
As you know.
There are no perfect world here.
But we decided to use Azure AD.
The live one.
But we don't use our production one.
We use Microsoft Office 360 developer program.
To acquire a separate.
Azure AD test environment.
And that is available for all developers.
In that one.
We can create all the stuff that we need.
In Azure AD.
To kind of get this scenario going.
And support it.
We create synthetic test users.
And disable the MFA.
There is no way around that.
But the good thing for me is synthetic.
It's not production.
It's like synthetic data.
All the way.
Our client component.
Will know that it's under test.
So now I'm challenging one of the test principles.
That I said was smart to have.
So try to make the test.
As little as possible.
But for you guys that are kind of known.
To the technical side.
What you are doing is to change the.
Underlying authorization.
From a code grant.
To a resource owner password credential.
And that is.
It's doable.
To serve the infrastructure.
We use Docker.
We put all these services.
The APIs in containers.
And we use Docker compose.
To kind of build them.
And serve the infrastructure for our test purposes.
We do the testing from the browser.
With the complete system running.
And we do that from the perspective of the end users.
And to do that.
We have Cypress.
Cypress from Cypress IO.
It's a really good tool.
I'm really starting to kind of love that one.
And what we can do with it.
And of course there are many, many more options.
The system.
The stuff that we create.
Should be kind of fully automated.
And then able to run in the pipeline.
So it should be able to run in GitHub actions.
If that's your treat or whatever.
That's one of the kind of.
That we have here.
So.
So exploring one potential solution.
And I say one potential solution.
Just emphasize.
This may not work for you guys.
It has been proven helpful.
For some of our teams.
And maybe you can take away a few things yourself.
So.
Running the application.
We are putting the system components into containers.
And now I will try.
The trick of sharing code.
This is how it will look.
When you kind of open the repositories.
We have the slide containing the stuff that you see here.
It's written in reveal.js.
It's a really good framework.
If you want to avoid the PowerPoints.
This is the scenario.
Here we have the client.
We have the diverse APIs.
And all the documentation and so on.
That's available to you guys.
But the containers.
We have a Docker compose file.
Describing a few services.
We have the client.
That is the application serving the front end.
And doing the actually.
Getting episodes and so on.
The second service is the episodes API.
And the third service is the quote API.
And we also created a service for Cypress.
More on that later.
You can see that there's a lot of environment variables.
That we use here.
And I have a few templates.
Just giving you an idea of what environment variables.
That are in use here.
So if this was a production scenario.
This is not how I would have done it.
I would typically append things like secrets.
And those kind of things.
Maybe even the whole configuration.
But this is how I would have done it.
In a keyword like Azure Key Vault.
Or similar stuff.
But for demonstration purposes.
This should be kind of sufficient.
So yeah.
And I have also in.
For those of you who are familiar with Azure AD.
Or any similar systems.
I have registered the objects.
For our systems in Azure AD.
And given the permissions.
That make this kind of run.
So I'm using Docker compose.
To run the application.
This is my application.
This is my command line.
I am now.
In the folder called scenario.
And I have a file called run.
What run does.
Is typically set the node environment to production.
And then do Docker compose up.
For the client, the episodes and the quotes.
So there's no magic to it.
But let's try to do that.
Seems to start okay.
I will.
Take off my glasses.
Open a private window.
Go to localhost 3000.
This is the super application.
That we have.
I do a sign in.
And I have to select the user.
I use one password.
As my password manager.
And using a password.
My password manager is always super smart.
I have this.
My test user.
On my own organization.
That I have in the office 365 developer portal.
Called cycling power.
I press next.
And then I select her password.
And I don't want to stay signed in.
But now I'm basically signed in.
To the application.
So show inbox.
Is getting me the office 365 inbox.
And as you know.
The whole process involved.
I have spoken to be able to reach the office 365 graph.
And so on.
Show got episodes.
Are hitting the episodes API.
That is hitting the quote API.
And suddenly it returns a list of episodes.
Along with a quote.
So that's basically it.
And of course some of the test challenges you see.
Is the login.
The authentication.
You know that one.
And it's typically where we kind of have the biggest problems.
So moving on.
To putting the system in test mode.
We use the node environment.
Variable called node end.
And we set the value to test.
Tell our system inform it.
That is under test.
And when we are testing.
We switch the authentication flow.
From code flow.
To the resource owner password credential.
And we provide the username and password.
To the backend.
So just to show a little bit about code.
How that is looking.
This is the main file.
The source folder.
The app.
And this is the login function.
So you can see that if my system identify is in their test.
It would use one specific function.
For authentication.
Another one if it's not under test.
There are better ways of doing this.
With dependency injection and so on.
But this works for the testing purposes.
And this is one of the changes.
For the testing.
The second one is the function doing the authentication.
Is basically this function.
It's in MSAL terms.
Creating a confidential client application.
And trying to authenticate.
With the username and password.
It got from the environment variables.
If all that kind of stuff is okay.
We have access tokens.
That is kind of built up.
In the token cache.
In the session cookie and everything.
So the only code that we have entered.
In our system to inform it.
That is under test.
Are this function and the three lines that you saw.
For the login endpoint.
Not too much.
So to run that one.
I will do the run T.
It's you'll see that.
With the client.
And so on.
But it tells the system that it's under test.
That's the important part.
I'll stop this one.
So it's up and running.
There's a lot more debug information.
Those of you are paying very good attention.
May see that I'm logging a bit too much.
Which is intentional.
So don't do that in the production system.
Again.
I'm in a private window.
Trying to log into the solution.
And as you recognize.
If I'm lucky.
There is no kind of authentication going on here.
Because that was handled on the backend.
And in the test driver.
I can manipulate the backend.
To log in with different users and so on.
And mimic various scenarios.
But the functionality is the same.
The inbox.
And remember.
Behind the scenes.
Everything is working as it would in production now.
Besides this log in functionality.
And the episodes more or less the same as well.
So just speeding on here.
When we're going to do the test.
We're going to use Cypress.
And I want to do a demo of Cypress.
And all the details.
I'll just show you how it works.
I'm running in the backend.
I open a new terminal window.
And then I go into.
Let's see.
Sorry.
Scenario.
And then the fronts are made a bit bigger.
To help you guys.
But this was not very readable.
But I go into the client.
Cypress.
And client is a node thing.
So what I typically do now.
Is NPM Cypress open.
And my computer will open Cypress.
The tool that I use.
To kind of define the testing.
The tests are defined in the client.
Cypress.
Cypress.
It's called integration.
And this is JavaScript basically.
Defining the test that we're going to do.
Log into the system.
Press this button.
Verify the output.
And so on.
More or less the same stuff.
That our users are going to do.
So what happens?
Maybe Cypress loaded.
Okay.
I will give it a little bit more time.
There it comes.
So the network now.
Is not running the DEFCON one.
It's running the Caesar resource.
And I have a VPN.
So the network is a bit slow.
I can double click this test specification.
The only one that I have.
And it will now open a browser.
Firefox.
The one that I have chosen.
Of course.
It complains.
Let me see why.
Let me try again.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
and the exit code you're going to kind of pass on is from
Cypress. So how this will work is we start up all the
services, Cypress will wait to execute until my system is ready
then it will execute and exit and Docker Compose will use the exit code
from Cypress. Which I typically then use in my build pipeline
to determine what to do, fail the build or what not.
So to do that stuff I do
I know that this doesn't make a lot of sense
just watching this kind of roll over but you can see the whole system is rolling
the Cypress is running all the tests and if you want to separate the logs
that's good. The essence is that I now run everything
concealed in my computer and all the tests kind of pass
for the whole complete system.
And that is typically what I would, what I was after.
So to start wrapping things up, the current status for our super complex
and modern application is that we now have a fully automated
system test scenario. All the test environment is synthetic, the users and so on.
I have a system with active authentication authorization, use a different protocol but it's
still based on the access token of all the other
things like the kind of in the later parts of the value chain.
All my security controls the stuff that I had in production is active
around my system and we have very little test support code. A few lines that I showed you to
drive a different login experience. So we have broken a few principles to reach this state.
The trade-off could maybe be kind of acceptable. Again I don't can't decide for you guys what I
really recommend is doing a threat model.
On your test setup, your contest environment and so on and make sure that you ask the question what
could possibly go wrong. Our objective with this kind of pattern was to encourage our teams to
divert more effort on to system testing with all the security features kind of activated. It's hard
not only a technical challenge but it's doable and in my humble opinion it's kind of necessary as well.
So,
that was a brief tour question and answers. I can leave this here while we do the questions.
Visit loop.ecu.no.com to learn more about what we do with software development and this is the
link to the GitHub repository with everything. So take a snapshot of that if you want to.
All right open for question if there are anyone. If you have similar experiences, done a bit
differently, whatever, please share. Very eager to learn what other people are doing.
Hang on a second. Is the mic on?
Is the mic on here? Just tap it.
Yes. It's on.
Is that better? Can you hear me now?
Yeah.
Have you had any success with trying to fake MFA so you don't have
to break an entire path to a user password?
So the question was have we had any success in breaking MFA?
You chose to go down the ROPC. Have you had any success with actually keeping with MFA with
automation?
No. We haven't really? We haven't gone and exhausted that route. We have some ideas that we're going to do. We've got some questions. But I'm going to go ahead and pass it to you.
could use but we we found this one to be yeah the one we selected and a bit better thank you
thanks for the great talk uh just a question about do you have any guidance for how to select
tests to system test your authentic like what's your methodology how do you pick your tests
and that's a really good question so the question was how what methodology do we do to do the system
test so i would like to say we do but we have like a hundred plus teams so they are all different
so what we advise the teams doing is is first like the develop teams on it owns what really
needs to work what what do we do to need to do smoke tests and all kind of things and then work
together with your business because these system tests are pretty much close to what the business
would write as accepting tests and that will be your kind of a that would be that
the goldmine. We have been trying to encourage the business to start writing Cypress tests
or explore that route but it's not sure if we will succeed but that will be like the
the golden goose in my head the business writing Cypress test that we just included us
in in a test suite so that will be beautiful so yeah. Thank you. One other question from a
developer to another developer. How do you make these tests more reliable because I know end-to-end
tests can be very fragile. That's a very good question. How do you make these tests more
reliable and i've been around for a while and I hated these kind of things because they were so
fragile so first time we did this we had to reference the point with an x and y and so on. So
no way Jose but after time went on and I started messing around with Cypress and similar
tools they are getting a lot better and I'm also doing a little bit stuff on tagging stuff in my
code making stuff out of what it is that we have done before. So I know that when we were running
easier to navigate and so on. So these days, when we do the examples, they're not that fragile.
There may be some hiccups in the network and so on, but not too bad actually. So I think it's an
option that more people should explore. Cool. Thank you.
All right. Enjoy the rest of the conference. Thank you.
Bye-bye.