Now, this morning, we have Wynne Schwartau talking to us about his history, which is really legendary.
So, sit back, enjoy, and please give a warm welcome to Wynne.
Am I supposed to ask for you to do that again to make me feel really good?
No, no, no, no, no.
This will fuck up the audio guy in the rear, right?
Am I fucking you up really bad?
This is going to be a rant.
I haven't spoken to DEF CON, I guess, in 15 years because I'm not the traditional hacker.
Network protocols, I lie, I cheat, I Google, and I have a lot of good friends.
A lot smarter than me.
But I began, and I didn't really realize that I was hacking.
And what I want to show you is some experiences that I had starting in 1958.
Anybody alive in 1958?
Fuck you, Barry.
Really?
Okay.
Oh, you were two.
And I grew up in an engineering family.
My father went to MIT at 50.
And worked on radar development during World War II.
He gave me a non-optional education on electrical engineering early.
My mother was the first audio engineer at NBC during World War II.
And got a whole series of patents on...
Yeah, my mother, I wish she could come to DEF CON, but she's like 140 now.
So, not going to work out.
That was my first DEF CON and my first beer.
And this is my father giving me...
Every week, my parents split up.
So, on the weekend, I went down to my father's house in Greenwich Village.
And watched the tourists get mad at the beatniks.
And he gave me non-optional electrical engineering education.
And that went on for years and years.
So, that ended up being weird.
But I ended up doing TV repair work at six years old.
Because how hard was it?
You take the tubes out of the TV, you go down to the drugstore, you test them.
You take the $2, get a new 12AX7, go back, plug them in.
And a neighbor in the apartment building gives me 50 cents.
But that was in the 50s.
And they thought, oh, how do you know how to do that?
Well, my father told me how to use the tube tester.
That was really it.
And that's what our electronics stores used to look like when we had to build shit.
And this is the world of computers way back then.
So, my first hack was in 1958.
At, anybody know what these are called other than Barry?
Rotary Telephones, of course.
And so we had these, obviously, and some of them were four-digit numbers.
And this is what a bill looked like back in those days.
When you actually had to pay...
rental fee for a phone and a rental fee for the wire. Phone calls, and this is in New
York City, were still 10 cents a minute even if I'm calling just across the street. So
things got expensive very quickly. And apparently I was an unruly child and talked a lot. So
well, I wanted to make phone calls. So what do you do in a non-destructive way? How do
you hack a rotary telephone with no tools, no destruction whatsoever? Well, here is the
old people up here and they're all giving me this. Now don't interpret that too sexually
please from them.
That's them. Inside of these are what's called stepper relays, decade relays. And when you
dial, basically it's click, click, three or a seven. And if you get your timing down with
the lock on the phone just right, hello operator, may I help you? And please connect me to this
long distance number in Greece or to my friend across the street. A non-destructive hack
that my mother then on the next bill came along.
And it was too much. And she goes, so you figured it the fuck out, did you?
Was it smart? No, it was just a kid not accepting that there was a lock that I could not bypass
even though it was a completely non-technical hack. So I'm screwing around with analog computers
and stuff throughout the years. And everybody know who Richard Stallman is? I guess he's
the guy that grew up with him. But I didn't remember that until I hit 60. Because we knew
him as Dicky, Dicky Stallman. And so he and I built little transistor radios and all sorts
of crap. And his mother was a MILF, just saying. I can't, shut up Ben, out. Now the
drugstores back in those days would hire a bunch of us kids for like three hours a
week each to spread a little bit of the 80 cents an hour wealth or whatever. And inside
of there, of course, you see, there's a phone booth. And those were the phones of the day.
And of course, we wanted to make phone calls. But you know, we're 10 years old. How do you
do it? How do you do it for free?
Well, that's the way these things used to work. Stick them in the slot. And this is
the US system. It was different. And you couldn't do this hack in other countries and not everywhere.
So here are the pieces to the hack. Chinese laundry cardboard and a penny. What we did,
and it was a design feature or fault by AT&T, depending upon how you look at it. You take
a sliver of the money. You take a sliver of the money. You take a sliver of the money.
You take a sliver of one of those cardboards. Slide it down the dime slot all the way to
the bottom. Pick up the receiver. Put a penny in the nickel slot. It registers as a dime.
And you get your penny back.
So if you have a pocket full of pennies, calling Greece with the long-distance operator
on the phone is a no-brainer.
And they're getting a legitimate money trigger. And we lived at that drugstore for years making
those kind of calls. And again, it was a stupid, stupid, but highly effective way
to bypass the security, the features, the controls that were built into that system.
So then I go on to high school, and they got a computer thing. And high school and
I didn't get along very well.
So I sort of left.
I left in 11th grade and decided to go into the family business, the recording industry,
music, TV, and all that stuff. And I was 16 years old. And so that was the first studio
I got to work at, which was fairly overwhelming, except for the fact that my father did it.
And I had been exposed to the studios, and it really wasn't all that scary.
Anybody know who that is? My first recording session. Shout it out. Anybody. Nugent. It
was the Nugent.
He would get naked in the studio and hump the console. Just saying. Please don't repeat
that and publicize that.
And then this was my first mechanical system. It was a cyber-mechanical system for cutting
records. Vinyl. Some of you may remember vinyl. And this is how we used to do that.
Now later on in life, I was at the studios. And there was a thing that was going on. I
would do this. This was a demand feature that used to be called telephone credit cards.
And there's one of them. And once you got the number, and you got the number by paying
a guy on 8th Avenue and 42nd street a dollar, you do that until the FBI decides to wrap
you around their heads a little bit.
So that was my first visit from the FBI for doing some sort of crime. That was 1969. It
was not a hack. It was pure out and out theft.
So then I get involved with Hendrix and all these guys doing complex systems, Robert Moog, and that's the first synthesizer down from Electric Lady Studios in New York.
So I kept seeing and being involved with complex systems.
This is where I learned that assembly language and I do not get along either.
And I built it, it semi sort of worked, and that was enough.
That was a computer that we used, DEC computers, to do synchronization for TV shows, and this one was from Applause.
And trying to make a DEC computer back in those days actually do what you wanted to do was an incredibly slow process.
Back in, this is the day in our studios in New York, we had our own telephone exchanges back in the day.
So how do you hack that?
Because what we needed to do was get audio from the network.
From the New York studios into the LA studios.
But AT&T wanted $1500 an hour in 1970 dollars to be able to do that connection.
But we had a switchboard.
And once you have the switchboard and the PBX system, you can bypass very easily the controls and get directly out to LA and move the audio back and forth.
Again, is it a hack by today's standards?
From our standpoint, it got the job done.
So being broke back in those days and a roommate, we used to get our utility bills on punch cards.
Did anybody ever do that?
Alright, one old guy.
Two old guys, fair enough.
Is that G-Mark?
Yes it is.
We'll take your mask off so I can see your face.
Thank you.
Yeah, let's hear it for G-Mark, come on.
Yeah.
So we used to get our bills and what did we want to do?
Not pay them.
And keep the power.
And the phones.
So we had access, of course, to computers at NYU, what have you.
And then we discovered by looking at the punch card, the dollar amount, let's say it was $7.32.
There were no fields to the left of the number.
So what did we do?
Add a minus sign.
Every month we got a check for the amount of service that we used the month before.
Wait a minute, is this doing a live thing of me?
Oh, shit.
I wonder if it can say that.
It will.
Oh, shit.
Okay, it did.
So that's how we, it worked for about two years and they upped their security a little bit after that.
So then I got hired to build a studio complex for some guys.
And while they wanted to pay me reasonably good money back in the day, they had virtually no budget for equipment.
And what we needed to do was tie together all of that.
So we were all of these studios throughout the facility, and there were eight or ten of them.
And instead of the big patch bay they couldn't afford, all the wiring associated with what they couldn't afford, we decided we have another option here.
What picture is this?
Yeah, AT&T.
AT&T owned the universe.
Them and IBM owned the universe back then.
And the AT&T building was about $5 million.
Five blocks away on 11th Avenue.
And what did we do?
We did what is now called dumpster diving.
Looking for old AT&T equipment that they threw out.
Because if there was one little thing wrong, they would just toss it out.
Which meant from our standpoint, all we have to do is find the one little thing wrong and fix it.
But it was all free.
So by putting all of this stuff together and repurposing this equipment,
we were able to build recording studios and switching networks using stepper relays the same way that we used to.
I hacked the phone when I was a kid.
And the stepper relay was the key to it.
Because you could dial up anything you wanted to do.
So there was a rack of dials and it rerouted all of the signals everywhere throughout the recording complex.
And it didn't cost us a dime.
Thank you AT&T for that one.
My first home computer where again I learned I'm a really shitty programmer.
I'm awful.
I got it to work semi-sort of kind of kludgy.
And that was when my wife first said that she was now a computer widow.
And that was in 78.
Oh, there's my wife.
And what happened with my wife...
Well, thank you, Mike.
My wife, she built recording consoles.
And this was...
This was the first computerized, automated recording console.
1979.
Never fucking worked.
I kept blaming my wife for it.
She said, I didn't wire those.
But it was one of these...
You had to know 700 control commands.
And do alt, phase, shift, antenna, metal.
In order to make anything attempt to work and it didn't.
And that was one of the projects that we were trying to make it work.
And...
Fuck it.
Just completely gave up.
I had a studio in New York for a while.
So that was the background on which we started...
Or I started learning.
There's always an answer.
It's like that Apollo...
One of the greatest scenes in movie history, Apollo 13.
This is what they have up there.
That's all you get to work with.
Nothing else.
So back in these days, I was doing a lot of studio work.
I was on the road with various musical artists.
And as we always say at DEFCON, something's always going to fuck up completely.
Well, I'm going to tell you about a couple of those fuck-ups.
And we had to hack our way out of them.
1975, 6, I don't even remember the year.
We were in Kingston, Jamaica.
And it was a thing that...
It was a thing with the British government and the...
They were trying to make peace because there were troubles down there.
And Stevie Wonder and Bob Marley were the big acts.
So we had to build riot cages into the stage
because of the suspected possibility of terrorism
that the British Embassy had told us about.
And remember, I think I'm 23 years old here going,
terrorism, concert, fuck me.
Get me home.
Well, concert goes on.
Soccer stadium.
120,000 people in the soccer stadium.
And keep in mind, the Jamaican police don't have police.
Ergo, the riot cages built into the stage.
So the chaos was suddenly...
All of the lights and all of the sound went off
with 120,000 ragging people in the audience.
The fuck do you know?
We're in a foreign country.
We don't know where the pow...
We have...
It was like, give us a plug.
It's like what we say.
Give me clean internet and I'll take it from there.
That's all we wanted.
Give us some good power.
Except in this case, there was so much power,
the pole transformer went.
120,000 people.
What do you do?
Well, one of the absolutely stoned out of his mind groupies...
It was his, not mine.
CO2 cools things down really quickly.
So we had two of the group...
Not the groupie.
Roadies, sorry, not groupies.
Roadies get the CO2s that we happen to have
because we had tube equipment and it gets hot.
Went over to start cooling down the transformer
as they start to get arrested
for terrorism.
And it took a few minutes
in order to get them to realize
we would like your help
keeping this...
But we hacked the entire power system
for the stadium
with several cans of CO2
and the concert finally went on.
Sort of like Jeopardy!
last night.
We finally got there.
Took a while, but we finally...
And again, is this a true hack?
I don't care what word you want to use for it.
It was when the shit
really fucking hits the fan.
What do you do?
And you've got to pull out
every single stop.
And I get into rants about graceful degradation
and enterprise networks and things.
And I talk to the CISOs and I say,
where's your graceful degradation and segmentation plan?
And they go, say what?
All right, well, you've never been in rock and roll, dude.
So there's a few takeaways
and the slides will all be available
for you guys later on.
I absolutely love Liza Minnelli.
We recorded Liza with a Z.
And what she did in the studio
was rather amazing.
She drank a bottle of scotch,
took her clothes off,
and proceeded to sing.
All right, that's the way you want to do it, Liza.
So the whole object
was going to be a TV show
called Liza with a Z.
She was coming back and doing,
you know, now going to be famous again.
So live shows,
live thing went absolutely fine.
But we need to go back to the studio.
Now, in those days,
we had film deck,
we had a video deck,
a little bit, but mostly film,
and an audio machine
with a thing called Pilotone.
Nobody's probably ever heard,
anybody ever hear a Pilotone?
Oh, Masucci, of course you have.
You're as old as I am.
I went to high school with this motherfucker up here.
Ninth grade, I've known him 56 years.
Thanks for coming, Mike.
And so you get back to the studio
and you expect the Pilotone
on the film and the mag stripe
to work with the audio, right?
Well, not so much.
Completely out of sync.
And we have a live broadcast
coming up on NBC.
The fuck do you do
when you're completely out of sync?
Well, you cobble together,
this is very sophisticated equipment
compared to what we have.
We cobble together
a set of oscillators and power,
two power amplifiers
to drive a set of
hysteresis synchronous motors,
for anybody that may remember what those are
before the DC motors came along.
And that Lissajous pattern,
the one that you see there,
we had to take dials
on the oscillators and the machines
because they introduced friction into the system
and maintain
that Lissajous pattern
for four hours.
By hand.
Did it always work?
Nah, sometimes the hi-hat
and the video was off a little bit.
But in general,
it was the best that we could do.
We cobbled together some
equipment that didn't exist
and we went to the shop and did it.
Again, is it a hack?
We got the job done when otherwise
there would not have been a TV broadcast at all.
Synchronization failures
were a disaster in the early days
of the recording industry
because everybody in the late 70s and early 80s
is saying, we're going to go digital.
Digital will fix everything.
Except for those of you who remember
what digital was like in the late 70s
and early 80s,
it was really highly experimental.
So we did a lot of experimentation
on how do you get
instead of just 24 tracks,
how do you get 48 tracks?
And can I record
24 of the tracks live in LA
the other 24
live in New York
and keep them all synchronized with equipment
that doesn't work from England?
No offense to my British friends out there.
Well,
synchronization was awful.
Remember the song
Do You Think I'm Sexy?
That was with Rod Stewart.
We were on that session for 72 hours.
72 straight hours.
Just trying to keep
the drum tracks
from that song
from LA in sync
with the audio
and the voice being recorded in New York.
That was 72 hours
of absolute hell
where we had to, again,
revert to graceful degradation
and a manual mode of working.
It was completely manual
and you could grab a few seconds of sync
and it would fall out again.
So it was an absolute nightmare.
But synchronization in our world
is about
what makes everything work
and getting the timing right.
We've all heard the stories of
well, my clock was on CST
and their clock was on Central European time.
We were eight hours off.
Or I'm using an imperial measurement system
versus a metric measurement system
and everything completely failed.
The synchronization issues
that we face
are very similar
to the types of synchronization issues
that we face today
but are much more profound
aspects of what can go wrong.
And that's why thinking synchronization
is so incredibly critical.
Charlie Daniels.
I got flown in from LA to record him
in Madison Square Garden.
My crew was there.
Everything should have gone very smoothly.
However,
what we were supposed to do
was record Charlie Daniels
and the event and the bands
through a fairly large console
onto a 16-track tape machine.
Fairly normal activity
for us back in those days.
So you have 40 microphones
go into the console
branch them out
and record them properly
on the appropriate channel
on the audio machine.
Except, and no offense to any union people
in this room,
but every day
Madison Square Garden
and the mafia
hired the most incompetent people
known to man
to assist us in our professional recording endeavors.
Every single microphone
signal was ground looped
so every single input
all 40 of them
had exactly
the same signal.
Right.
No isolation.
No differentiation.
And we actually had to do it
all live
with massive amounts of equalization
and digital filtering
such as existed in 1981
to be able to get it down
to a mono machine.
And then have to go to the studio
and rebuild the whole thing from scratch.
So audio quality
if anybody's interested
I actually have the vinyl from that.
The audio quality
is absolutely awful.
But the radio stations
didn't seem to give a shit
because they were broadcasting on AM
and the quality was so bad anyway.
So this was about preparation
and architecting
and never really trusting
folks to do it
do the jobs that you expected
especially in that environment
back in those days.
So there's plenty of realities
and I spent a lot of time doing remote work
and remote work is where
you're fucking wrong.
And it's gonna go wrong in a big way.
And these are just a couple of the examples.
There was a studio
we worked at called R1
and it was a jingle studio
back in the days
the money making for the recording industry
that would support artists
was from the jingle houses
for advertising.
And so it was pretty straight ahead stuff.
And so we had
the studio was roughly like this
and 40 musicians
out there, the producer
from whatever company it was
I guess this one was Alka-Seltzer
and the jingle house
they're all sitting there
and then they say to me
When?
Is there supposed to be smoke
coming out of the console?
Oh fuck!
So we got 40 guys
and we got a console
that just went up in smoke.
What do you do?
What do you do?
Well you panic.
And back in those days
we had these things
and I think some of the AV guys still use them
they're called SM
they're mixers
little tiny Shure 4 channel mixers
and so what did we have to do?
We had to get all the gophers in the studio
extend all the microphones
wires about 100 feet
wire them into the studio
put them into the back of these little mixers
take those mixers
bypass the console
put them into the machine
where we can manually control the levels
throughout the entire session
and never be able to hear
what was going on
until we did a playback
to see how good it was
that was sheer chaos
but again
there was a way to do it
we understood the architecture
of what we were trying to achieve
we understood
what we had in the consoles
how do we replicate
what the future set
into something else
in under 20 minutes
and it's a bitch
but these are the things
that have brought me
into the security industry
always thinking of
what the hell can go wrong
and maybe that's why I wrote
several of the books
on these topics
of what can go wrong
when you don't really think about it
Charlie
Hank Williams Jr
we were doing a thing in New York
anybody ever hear of the Lone Star Cafe
in New York
we had a studio in the basement of it
and every week
we'd record
whatever the act was
live
to 570 radio stations
across the country
not really a big deal technically
we knew what we were doing
we had the tie lines and everything went well
Charlie Williams is up there
and he's playing
and a couple of guys
whose particular orientation
he was not thrilled with
decided to have a beer fight
and throw beer on him
on stage
it's a little tiny stage
smaller than this
and he up and walks out
and we hadn't been through 30 minutes
of the live show yet
gone
no act
what do you do
what do you do
well we had the radio announcer
Mike Fitzgerald
was his name
he was the New York radio announcer
for the station that sponsored all this
and I said Mike vamp
vamp
and I said I need 10 minutes
so what did we do
we were live recording everything
I rewound the tape
to the very beginning
and we introduced him live
and pressed play
and replayed
for the second half of the show
exactly what was done in the first half of the show
nobody seemed to give a shit
it was
again
back in those days
and I think it's still the same rules
if you don't fill the air time
the FCC is going to get on your ass
so whatever it was
fill the air
fill the air
the live stuff
when you're doing
any technical stuff
at all
and you say
oh this is going to be really easy
the answer is no
how many years of hacker jeopardy
have we had
where we
ah we got it wired this year
until somebody has
decides to put vista on a machine
for us
and it was the first time
I think anybody ever yelled at jeopardy
hacker jeopardy was
hey does anybody have a non vista machine
we can have now
and it finally worked our way through that
as we build
our systems
complexity is
our enemy
absolute enemy
and we want more features
so we see the
increase of
sophistication of software
we see the increase of sophistication of hardware
not necessarily so much because we need it
but the vendors
want to be able to say
we have 22 new features
and it's worth another thousand dollars
for us to give it to you
and we're increasing
our complexity
without due
thought
without proper planning
again
what do you do when the IT hits
the fan
and that's why I go back to these presentations
and my prior career
of it was all analog
we had nothing digital that worked
in those days at all
with the exception of a couple little pieces of equipment
everything was guessing
trial and error
and it was a mess
and today we're still doing this trial and error
but we should learn from history
that a little bit more planning
in when the IT hits the fan
how do you function
what is your minimal viable
operating capability
do we plan about that
the businesses go IT you take care of
the business folks don't understand enough
you guys do
and that is part of the
hopefully the lesson here
is to be able to plan for
absolutely everything
one of the things we used to do back in the day
get a recording console
let's say a new recording console is in
or we've done an overhaul
of a particular part of a studio
one of the things is we called it
turn all the knobs
what we did and the reason we did it
we wanted to crank up
the volume in every single
circuit on the console
and we're looking
4000 knobs maybe
on a good console
because we wanted to stress it to the point
we wanted it to break under
our lab conditions versus
break in a live situation
a TV situation or
live recording in the studio
so cranking all the knobs
for us helped
identify problems similar
to where airplane engine
design is done stress
the hell out of it
and one of the things that I think I've missed in our
field and if somebody
is doing it really well I would love to know
about it is do we have
any sort of protocols for
stress testing of
our networking infrastructures
to be able to say okay
we need to put a limiter
or a compressor or a restriction control
because it cannot handle
all the traffic all the
things so back in
91
we were building
Steve Katz the world's
first seesaw brilliant guy and a dear friend
from
JP Morgan they wanted to do the first
single sign on
system 1990
and I was
on that team and
we tied together security dynamics
which became RSA
there was Sun Microsystems
with Solaris
the
Delphi project from MIT
was involved and so
we built this whole single sign on
system we were
so proud of ourselves so
proud we're showing it at a couple
conferences and it works
brilliantly we're
single sign on is finally a
reality until
it fucks up
so Steve says let's
deploy it deploy it where
well we're going to do it at JP
Morgan cool
they deployed it
company wide
sounds
good right
well what we had not
considered was
the time factor of
authentication when
10,000 people try to log
on at 9 a.m. at the same
time
what happens to your network
when you don't have
accelerators you don't have
spread bandwidth
you don't have balancing circuits
all the shit we have today
we didn't have and we completely
fucked it up by spending
all this money in designing something that was
great for 10 people
not for 10,000 people
so turning all the knobs
stress the shit out of things I
cannot impress enough
that we need as an industry to really
develop a stress mechanism
a stress protocol for
various of the environments that we're in
and it's turn all the V knobs we can
do the same thing and we should
be able to very easily automate
that stress
testing process and I see
Dave fizzes up here he's designing
it right now do you have the patent shift for this
yeah
so
incident response whether
it was the incident response with my mother
and the rotary telephone on my
and the response was fuck that I'm going to make a
phone call somehow
all the way up to catastrophic
failures with
a couple hundred thousand people
live TV with all
of those and I didn't even show you some of the really
nasty ones it's all
about incident response and
appropriate planning
understanding your architectures
and then making sure that you have a
way to degrade
and a methodology to recover
and on that note
I wanted to thank you so much
I'm here for some questions if you like
but that was my hacking career
beginning in 1958
I am still a really
really shitty programmer
I have no idea how to do it anymore
and you guys have been wonderful
hope to see you tonight at Hacker Jeopardy
and if you have any questions or comments
there's a goon with a microphone
out there
oh
thank you
all right
if there's no questions again thank you
very much see you tonight at Jeopardy
I appreciate it all