All right. Hello, everyone. So our next speaker is working at Body Hacks, the body hacking
convention. And his focus is to introduce hackers to the concept of sensory augmentation, basic
science behind our senses, and neuroplasticity. So here to present his talk, Hack Your Senses,
Sensory Augmentation and Neuroplasticity, ladies and gentlemen, Trevor Goodman.
Wee!
There you go.
Thanks.
All right. So like he said, this is Sensory Augmentation 101, I think, on the schedule,
or I renamed it at the last second.
And definitely.
Hack Your Senses, Sensory Augmentation and Brain Plasticity.
I'm Trevor Goodman.
I'm the events director for Body Hacks, the body hacking convention in Austin, Texas.
If you want to ask questions afterwards or anything else, the QA mic's great.
But if you forget that, this is me on Twitter, at Bello Osa.
Let me put my notes back up.
Don't mumble to yourself when you have a microphone on.
So body hacking is about using and applying techniques and technology to make your brain
and body better, and this includes technology and techniques that are old as well as ones
that are new.
So you'll see both referenced here, but today I'll be focusing on hacking your senses specifically,
because your senses are the only mediator between your mind and the outside world, so
that includes your body.
So I believe sensory augmentation has the most potential for helping us really understand
our environment.
But what is sensory augmentation?
This is Neil Harbison here on the right.
For those of you who don't know him, Neil was born monochromatic, so he's completely
colorblind.
He sees in black and white.
But while he was at university, he worked with an engineer to develop this antenna that
is osseointegrated into his skull that lets him hear through bone conduction in a wider
range of colors than you can see.
So Neil's one of the most well-known people to have augmented senses, and is certainly
one of the more dramatic cases.
But there are likely people here at DEF CON who have magnets in their fingertips, or probably
some people who have cochlear implants.
So sensory augmentation is extending the body's ability to sense.
The body's ability to sense aspects of the environment that are not normally perceivable
by your natural state.
So you're born, anything you do on top of that is sensory augmentation.
For the most part, sensory augmentations currently rely on using sensory substitution,
which is using one sense to replace another.
We'll touch on a lot of sensory substitution as we go along, because the vast majority
of sensory augmentation right now relies on that.
Direct brain interfaces are being tested right now at DARPA, and DARPA-related re-net projects,
including one at Case Western University, where they've linked a prosthetic to a sense
of touch directly into a guy's brain.
So if he's rummaging in a bag or picking up objects behind a curtain, he can have
some sense of what he's doing, even though it's through a prosthetic.
There's another related project through re-net, where DARPA has hooked a paraplegic woman
up to a flight simulator directly through her brain, and she can fly the jet using her
brain, but I'm not sure if they have the jet sensors feeding data into her brain, or if
she was just using the monitor.
But we're really close to getting into some of that space.
Is this me?
Is this it?
Okay.
All right.
But how does that work?
How can we essentially just plug electrical current into our brain and get our senses
on the other end?
Because that's crazy, right?
So neuroplasticity.
That's how that works.
This is Paul Bakirida.
His father had a stroke in the 50s, and when you had a stroke in the 50s, the doctor said,
you're screwed, you'll never walk again, sorry.
But his brother, George...
George was a psychiatrist, and he felt like his dad could walk again, so he made him crawl,
he made him go through all this really intense, horrible physical therapy for the time.
And all his doctors said, you're crazy, you'll never walk again, this is stupid, why are
you doing this?
But eventually, he did walk again, and now we know that's what we do with stroke patients.
We make them go through really horrific physical therapy until they can relearn to walk again.
And after his father's death, they did an autopsy and found that the stroke-damaged
areas of the brain were still there.
So it's not like they're just gone.
So it seemed like this was evidence for neuroplasticity, and that the brain had rerouted its connections
around the damaged areas.
And that's the wrong button.
Paul Bakirida went on to study neuroplasticity by creating sensory substitution devices.
One of those devices originally was built to teach a sense of balance to people who
had lost it in injury.
Eventually, it became the brain.
The brain port, I think this is the original balance sensory substitution device, but it
may also be an early version of the brain port.
The brain port takes data from a camera.
So what it does is it's intended to help blind people have a sense of their environment.
So it takes data from a camera, it transmits that, or changes into a gray scale pixel grid,
and then it takes that gray scale pixel grid and transmits it to an electrode matrix on
your tongue.
And so you wear, like, you have this little rubber thing with a bunch of little electrodes
on it, and you stick it on there, and it vibrate.
And it's very powerful.
And it buzzes on your tongue in intensity based upon how dark or light the colors are.
And with modern brain port and proper training, a new blind user can safely navigate a brand
new environment in less than 10 hours of training.
And so now we can see on brain scans that activity is registering in a visual cortex,
even for people who don't have sight through their eyes.
So Paul Bakirida always said, you see with your brain, not with your eyes.
As Tim Cannon said first thing this morning, using these exact same words, you see with
your brain, not with your eyes.
These exact same words, which I thought was lovely.
Your brain is a pattern recognition machine.
That's mostly what it does.
It takes in sensory data in the form of electrical signals through your eyes or your touch or
any of your other senses, and then layers it on top of your expectations of your environment.
So your brain supersedes your senses.
So you see with your brain, not your eyes.
But your senses feed back into that and develop predictions and expectations of your environment
over time.
So you have this feedback loop, even though your brain tops down your senses.
And we know your brain tops down your senses.
Because of stuff like optical and auditory illusions, your brain is trying to figure
out and guess what's happening here as your eyes are confused.
So we know we can augment our senses.
But why would you do that?
So this is my snail friend.
If you were a garden snail, you'd be able to taste and you could smell.
And you'd be able to feel your body balanced over a leaf.
We also know that snails can traverse over the sharp edge of a knife blade without cutting
themselves.
So they have some sense of body position.
And your little, cute, cup-shaped eyes would be able to tell the difference between light
and dark.
And you could see things a few inches away.
But the snail's sensory bubble is pretty small.
The part of the world it can know and can understand is tiny.
As most of you are human, we've got a wider array of senses than that.
They're commonly broken up in five.
But it's really easy to identify more than that.
Hearing is an obvious one.
Smell and taste go together.
Vision is, we just call it one sense, but it's also light and color and edge detection
and motion.
So there's lots of other things that fall under your visual sense.
Touch comes into pressure and pain and cold and heat.
All of those get lumped in there.
All of those are your somatic senses.
And then you've got a whole host of inter-receptive sensors for things like blood pressure, lung
inflation and hunger.
And balance and joint position go into those things, too.
So if you don't...
If you don't understand what all of these are, for instance, we'll talk about joint
perception.
It's one of my favorite things that you don't think that you have it as a sense.
But if you close your eyes and move your arms around and you stop and you think about
where your arms are in space, then you know where those are.
So you have some sense of your body position that's separate than anything else.
Loss of proprioception, which is the position thing, that's what we call it, it's a stupid
name.
We should just say position.
It's a rare disorder.
But it can cause lack of cordiality.
It can cause lack of coordination and eventually complete lack of awareness of the body.
So it makes you feel disembodied as if, like, you weren't really in your body.
And this is also what the police test when they think that you're drunk and they're
making you touch your nose or walk a narrow line.
All of these things together are what we call perception.
German philosophers would call this your umwelt, but we'll stick with sensory bubble.
I think it makes a little more sense for most people.
You're probably pretty happy with these amazing human senses that you've got.
Instead of the snail senses.
But there are a lot of things other animals can perceive that you can't.
So the visual range, let's just talk back about sight, because we're mostly sight-oriented.
The visual range that we see is a tiny sliver of the electromagnetic spectrum.
That includes everything from gamma rays and x-rays on one side to AM, FM, and microwaves
and cellular on the other end.
So imagine for a moment all the different types of radio signals moving through this
room.
And you just can't see it.
You don't process them.
They're here.
You can pull out a tool to do that, but you can't understand it.
But if you look just outside of our visual ability, you'll find ultraviolet and infrared
in a lot of animals already.
So bats will do echolocation.
Snakes can see infrared, but it's a molecular sense.
It's actually not visual-based.
So it's a chemical thing.
And then bees can also see in ultraviolet.
Also Neil Harbison can see in ultraviolet, infrared.
All right.
So it's not outside of what we can do.
Additionally, even if you just want to look at color, there are a lot of other animals
that have different color cones, more color cones than we have, and who can differentiate
between different wavelengths better than we can.
So like, oh, we'll talk more about that in a second.
Catfish are fun.
They've got about 250 times as many taste receptors as you do, and they're all over
their skin.
So they taste their environment as they swim through it.
And that is an entirely different sensory experience of their world than we have.
Imagine tasting as you walk through DEF CON.
For dogs, this is a good dog.
He's also wearing AR glasses.
And they, I mean, he's already augmenting his senses.
I mean, it depends on what software he's running.
But they possess up to 300 million olfactory receptors, which is compared to 6 million
in us.
So.
As we are visual, we understand our environment visually.
Dogs understand their environment through their smell for the most part.
So you can blind a dog and it really doesn't lose as much of its understanding of its environment
as we do.
But it's seeing chemical trails pass through its environment all the time.
But we don't stop with sensors inspired by other animals.
We can build senses that let you sense the ozone level in your area.
Moon Rebus has an implant in her elbow.
That lets her feel earthquakes as they happen all over the world.
You could build something that lets you know the amount of email in your inbox or stock
prices or sense other cyborgs walking down the hall.
Personally, I've got a horrible sense of direction.
So I've always been interested in stuff like the bees and a lot of birds have a magnetic
north sense.
Oh, before we go anywhere, Narwhal is really cool.
We skipped over this.
But so this is actually a tooth.
It's not a horn.
It's a horn.
And that tooth senses salinity and temperature and water pressure.
And so like they're sensing a lot of stuff about the salinity is really cool because
they breathe air.
So when they're able to sense it's something about when the when the water freezes on the
top of the ocean, the salinity changes.
And so they can tell when the ice is frozen above them or not.
So if they need to get up for water, they know where they can air.
They know where to do that.
So here's a fun example.
This is a compass.
You'll probably know what those are.
You hold it in your hand and it tells you where magnetic north is.
This on here on the right is North Sense.
It was designed by a company called Cyborg Nest.
They released it about six months ago.
Their mission is to create new senses.
On the board of that project is Neil Harbison.
Also they've got some body modification experts in there as well.
So this is a nice example.
Steve Hayworth.
That's who it was.
Steve Hayworth is also on this project.
So at face value, oh, sorry, Cyborg Nest's mission is to create new senses for humans.
And it's pierced on your chest.
So it sits right here.
And it's I think it's about one centimeter by two centimeters.
And it vibrates when you face north.
So at face value, both of those are really similar.
You're like, why?
Why would I do this?
Why would I put this thing on my chest when I can just use a compass?
But the compass is a tool that relies on your constant focus and attention.
And that's why it's called a compass.
And north sense is designed to be worn and felt all day every day.
It's designed to be slept in it.
And so over time, these become part of your experience.
And it blends into the background like the hum of the AC or things happening in your
peripheral vision.
So it starts to sound a lot more like how your senses work than how compasses work.
Because a lot of the things that you're processing with your senses are processed in your unconsciousness.
And until something pops out and stands out, that's unusual.
So if it went bah!
Now everybody's paying attention.
But if somebody made a noise in the back, you might not even notice.
So here's the thing about the brain.
The thing it does best, again, is to see patterns in data.
And so when you put that sensory data on your chest every day, and it creates a predictable
pattern in your environment, so every time you face north, it buzzes and tells you that,
you start to feel and dream and think in north.
You have attachments of memories relating to where they were facing.
What direction?
What direction they were facing on the planet.
And NorthSense has actually got enough users, finally, that they're starting to work with
I don't remember the university, but there's a university study with NorthSense users that
are doing an N of about 150.
So they're going to get some decent data, and they're going to be able to see how people
are processing this and what it's doing to the way that they interact with their environment.
So again, I argue there's a huge difference between these two things.
This is the big argument on the internet whenever somebody talks about NorthSense.
Well, I'll just get a compass.
Other big projects in this space that I think are important and really help do this, this
is NeoSensory's vest.
This is one of my favorites.
It has vibrator motors attached to the side here, and the vibrator motors translate sound
waves into tactile feedback that are intended to give people who are deaf a way to process
sound.
So if you're wearing the vest and I say, technology, it will go .
And if somebody else says technology, it will take those same sound waves and it will go
.
So it's not taking the words and then converting it into tactile feedback.
It's converting the sound waves themselves.
And this is at the wormhole.
So we're playing with it here.
And you can see it light up as it vibrates the music.
But the applications don't necessarily stop with playing at dance parties and helping
deaf people hear.
The vest can be set up to translate stock data into feelings so you can feel the dow
fall and plummet all day long.
.
Or to feel the pitch and roll and yaw of a drone.
So if you're wearing a VR headset and you're wearing a vest and you're flying a drone,
you're going to have a much more I'm in the drone experience than you're going to have
I'm sitting here watching it on the ground with a thing.
In the middle, this is an AR helmet.
I don't actually consider this sensory augmentation, but I think it's an important example.
This gives a heads-up display to the motorcycle riders.
And it's definitely more wearable tech.
But as some of this stuff gets better, you can certainly see how if you took this augmented
reality and you applied it to your vision constantly, how it would become a part of
your subconscious reaction to your environment.
This on the right side is Daniel Kish.
So you don't even need really cool technology.
He takes groups of blind people out in the wilderness and trains them to use their hearing
to echo-lacate using clicks or cane-tips.
And there are other projects that are using gene therapy to add red-green cones to people
with color deficiency.
It works in chimpanzees.
And others working on making people tetrachromat, so having a yellow cone.
But this is DEF CON.
So if you can't do it yourself, why do you care?
So here's some examples.
This real quick is, I think it's a Northpaw or Southpaw.
This is a variant of another North sensing device.
They built this based upon a. . .
Project at a university where they had built a belt with vibrator motors all around it
and it would vibrate where North was all the time.
They saw similar results in their stuff as North senses seeing.
This is the same thing.
It's on a smaller scale, so it's like a little prison bracelet on your ankle that vibrates.
We've already talked about magnets a lot today, so we'll not do too much of that.
But there are a lot of other projects, Bottlenose for instance, that is supposed to go over
that and help you see this.
Haptic sense, haptic stuff is really interesting.
This is, again, another way you can do ultrasound.
This is an Arduino project on Instructables.
This here is the voice, so there are a couple different apps even.
This is a different visual program.
This is like the brain port, but instead of translating it into an electrode grid, what
it does is it scans vertically every half a second or second and it gives you an audio
signal of what's in front of you.
That's an Android app.
It's also for iPhone.
It's also for Windows.
You can turn it on.
It's the voice for Android.
You can download it and start playing with it right away.
I've got it on my phone if you want to see it.
These are all projects from somebody that is really awesome.
She does a lot of research in this space, so she has a lot of really fun projects.
Her name is Ison.
This is kind of like the brain port.
This is her version of trying to recreate the brain port.
This has a much lower resolution than the brain port does.
She's also done an ultrasound.
That was intended for...
For being underwater.
This is another echolocation project.
She found that if you blind people, they actually start to use their visual cortex really quickly
and adapt to it a lot more quickly.
This is a watch that gives you different smells depending on the time of day, so you can get
a time sense that's substituted with your smell.
And then this is a bone conduction headset that she calls the Play-A-Grill that has an
MP3 player that plays through your teeth and then through your jaw.
Okay.
Where was the other thing I was going to say?
I have another cool thing you need to hear about, and I forgot it.
Oh, yeah.
So there are a lot of different groups that are putting some of these different projects
together.
There's Hack the Senses in Budapest.
There was also recently Cyborg Futures at Parsons, where they're putting together workshops
to help people design and build new senses.
I'm hoping to see some more stuff like this at Grindfest soon, but I hear they're mostly
interested in shocking each other.
And with open source hardware...
Oh, where was the other thing I was going to say?
Like, there's a company called Bitolino and OpenBCI.
Both of those do different biosignals.
And the basic stuff you can get at Adafruit and SparkFun, you can do a lot of fun sensory
stuff right now.
So if you want to hear more about this and related stuff, here's my stupid pitch that
everybody hates.
Come to Body Hacks, first weekend of February, Austin, Texas.
Here's how to reach us.
Here's me.
There's my thing.
Questions?
I got like two minutes, right?
More than...
All right.
Cool.
Who wants to be hit with a squishy brain?
Cool.
Any questions?
Questions?
Nope.
Cool.
Yeah.
You can ask me here if you want to talk about it later.
Oh.
Go ahead.
Okay.
Thanks.
So you had about 20,000 subscribers.
And you have 20 or so human senses listed, which is different from the five senses.
Yes.
Which one does the fifth sense sound fall under?
Is it pressure?
Sound?
Yes.
Sound is its own thing.
It comes in compression of air pressure waves into your ear.
So it's...
So it's totally its own thing.
So it's a type of pressure.
I guess it's a type of pressure reception.
I'm a lay expert in this stuff so that may be over my head.
All right.
Thanks for having me.
Yeah, I mean, it bounces off your eardrum.
Thank you.
No worries.
So for the RFID chips, I know that there is documentation out there
that you give to a piercer or a body modification expert,
and they are pretty easily able to do it.
Do you know if the NorthSense is the same, or is that a lot more difficult?
So this is funny, actually.
I was reading some stuff on it this morning.
There is a lot of disagreement about whether the NorthSense,
and it's mostly in whether it's a good place on the chest or not,
because a lot of piercers will say, hey, it's pierced on transdermals.
So they were using the round ones, but they've moved to flat transdermals,
and NorthSense says that they think that those are better,
and they haven't had any issues in the people who they've been testing with.
But people like Sampa Von Cyborg are like, hey, well, let's just wait and see.
So part of this study of 150 people is, I think,
hoping to figure out whether it's an actual good test location.
And I'm really excited about this stuff,
but I'm super worried about knocking all that and hitting it
and having them come off.
So personally, I don't think that's a good location for me for that,
but a lot of people are trying it, and we'll have to see.
Are they going to place it on, like, a back?
Has anyone tried anything else?
I don't know for sure where other locations have been posted.
I know there's a guy, Russ Fox,
who I think has placed them in a couple other places,
but I don't know specifically.
Thank you.
I know those flat transdermals are a lot better for piercers.
Yeah, well, that's what they're saying, too, but we'll see if it works.
All right.
So how do you explain all this to your doctor?
How do I explain?
I mean, I'm lucky enough to not really get sick.
But, like, what questions would your doctor have?
For example...
You be the doctor, I'll be you.
All right.
So why did you decide to pin this magnetic north sensor to your test?
Oh, well, okay.
In that kind of case, I like to ask that with another question.
It's like, okay, well,
if you could sense more of your environment around you,
why would you not want to?
All right.
Second question.
You need an MRI.
Do you have any magnets on you?
Yeah.
Do you have them in your fingertips?
Yeah, well, I mean, so the fingertip magnets, I think,
it was funny, we actually talked about this at the Q&A for the last talk.
There's a lot of argument about whether it actually causes issue with MRIs or not.
I don't have any personal experience, but with, I mean, the north sense,
you could unscrew the balls and pull the thing off if you wanted to for an MRI,
and hopefully you're not getting MRI every week.
And, I mean, most people probably don't get them every year.
So I think for those kind of things, those are niche issues,
which we have issues with any technology.
So I don't think it's any different than any other thing that we adopt.
All right.
We adopt.
Last question.
So a couple years ago, I saw a demonstration from Tachi Lab on YouTube.
They had some texture-based stuff.
They had a glove.
With a little thing on the inside, and you would touch a touch screen,
and you would feel the synthetic texture on.
Nice.
So they're doing a haptic thing with the.
Right.
So then you've got like Senseg, right, which is the, or Sense-G or whatever it is,
where you have to drag your finger across it.
So everything I could find, and they're all Japanese,
so I tried emailing them, but they couldn't respond to me.
Like, how do you get that fine-grained texture thing without the movement?
I don't understand the science between a lot of that stuff.
And you're talking about projects that I actually haven't heard of.
So we've had a lot of trouble reaching people in Asia for stuff like this until recently.
And so a lot of that is just now starting to change.
Those lines of communication are opening up.
So I'm not familiar with those projects.
So please tell me about them and give me links so I can go look at them.
Okay.
Cool.
Yeah, my background is in running events and psychology, so.