Jeff Hawkins: How brain science will change computing

Jeff Hawkins: How brain science will change computing


I do two things: I design mobile computers
and I study brains. Today’s talk is about brains
and — (Audience member cheers) Yay! I have a brain fan out there. (Laughter) If I could have my first slide, you’ll see the title of my talk
and my two affiliations. So what I’m going to talk about is why
we don’t have a good brain theory, why it is important
that we should develop one and what we can do about it. I’ll try to do all that in 20 minutes. I have two affiliations. Most of you know me
from my Palm and Handspring days, but I also run a nonprofit
scientific research institute called the Redwood Neuroscience
Institute in Menlo Park. We study theoretical neuroscience
and how the neocortex works. I’m going to talk all about that. I have one slide on my other life,
the computer life, and that’s this slide here. These are some of the products
I’ve worked on over the last 20 years, starting from the very original laptop to some of the first tablet computers and so on, ending up
most recently with the Treo, and we’re continuing to do this. I’ve done this because
I believe mobile computing is the future of personal computing, and I’m trying to make
the world a little bit better by working on these things. But this was, I admit, all an accident. I really didn’t want to do
any of these products. Very early in my career I decided I was not going to be
in the computer industry. Before that, I just have to tell you about this picture of Graffiti
I picked off the web the other day. I was looking for a picture for Graffiti
that’ll text input language. I found a website dedicated to teachers
who want to make script-writing things across the top of their blackboard, and they had added Graffiti to it,
and I’m sorry about that. (Laughter) So what happened was, when I was young and got out
of engineering school at Cornell in ’79, I went to work for Intel
and was in the computer industry, and three months into that,
I fell in love with something else. I said, “I made
the wrong career choice here,” and I fell in love with brains. This is not a real brain. This is a picture of one, a line drawing. And I don’t remember
exactly how it happened, but I have one recollection,
which was pretty strong in my mind. In September of 1979, Scientific American came out
with a single-topic issue about the brain. It was one of their best issues ever. They talked about the neuron,
development, disease, vision and all the things you might want
to know about brains. It was really quite impressive. One might’ve had the impression
we knew a lot about brains. But the last article in that issue
was written by Francis Crick of DNA fame. Today is, I think, the 50th anniversary
of the discovery of DNA. And he wrote a story basically saying,
this is all well and good, but you know, we don’t know
diddly squat about brains, and no one has a clue how they work, so don’t believe what anyone tells you. This is a quote
from that article, he says: “What is conspicuously lacking” —
he’s a very proper British gentleman — “What is conspicuously lacking
is a broad framework of ideas in which to interpret
these different approaches.” I thought the word “framework” was great. He didn’t say we didn’t have a theory. He says we don’t even know
how to begin to think about it. We don’t even have a framework. We are in the pre-paradigm days,
if you want to use Thomas Kuhn. So I fell in love with this. I said, look: We have all this knowledge
about brains — how hard can it be? It’s something we can work on
in my lifetime; I could make a difference. So I tried to get out of the computer
business, into the brain business. First, I went to MIT,
the AI lab was there. I said, I want to build
intelligent machines too, but I want to study how brains work first. And they said, “Oh, you
don’t need to do that. You’re just going to program
computers, that’s all. I said, you really ought to study brains. They said, “No, you’re wrong.” I said, “No, you’re wrong,”
and I didn’t get in. (Laughter) I was a little disappointed —
pretty young — but I went back again a few years later, this time in California,
and I went to Berkeley. And I said, I’ll go
in from the biological side. So I got in the PhD program in biophysics. I was like, I’m studying brains now.
Well, I want to study theory. They said, “You can’t
study theory about brains. You can’t get funded for that. And as a graduate student,
you can’t do that.” So I said, oh my gosh. I was depressed; I said, but I can
make a difference in this field. I went back in the computer industry and said, I’ll have to work
here for a while. That’s when I designed
all those computer products. (Laughter) I said, I want to do this
for four years, make some money, I was having a family,
and I would mature a bit, and maybe the business
of neuroscience would mature a bit. Well, it took longer than four years.
It’s been about 16 years. But I’m doing it now,
and I’m going to tell you about it. So why should we have a good brain theory? Well, there’s lots of reasons
people do science. The most basic one is,
people like to know things. We’re curious, and we go out
and get knowledge. Why do we study ants? It’s interesting. Maybe we’ll learn something useful,
but it’s interesting and fascinating. But sometimes a science
has other attributes which makes it really interesting. Sometimes a science will tell
something about ourselves; it’ll tell us who we are. Evolution did this
and Copernicus did this, where we have a new
understanding of who we are. And after all, we are our brains.
My brain is talking to your brain. Our bodies are hanging along for the ride, but my brain is talking to your brain. And if we want to understand
who we are and how we feel and perceive, we need to understand brains. Another thing is sometimes science leads
to big societal benefits, technologies, or businesses or whatever. This is one, too, because
when we understand how brains work, we’ll be able to build
intelligent machines. That’s a good thing on the whole, with tremendous benefits to society, just like a fundamental technology. So why don’t we have
a good theory of brains? People have been working
on it for 100 years. Let’s first take a look
at what normal science looks like. This is normal science. Normal science is a nice balance
between theory and experimentalists. The theorist guy says,
“I think this is what’s going on,” the experimentalist says, “You’re wrong.” It goes back and forth,
this works in physics, this in geology. But if this is normal science,
what does neuroscience look like? This is what neuroscience looks like. We have this mountain of data, which is anatomy, physiology and behavior. You can’t imagine how much detail
we know about brains. There were 28,000 people who went
to the neuroscience conference this year, and every one of them
is doing research in brains. A lot of data, but no theory. There’s a little wimpy box on top there. And theory has not played a role
in any sort of grand way in the neurosciences. And it’s a real shame. Now, why has this come about? If you ask neuroscientists
why is this the state of affairs, first, they’ll admit it. But if you ask them, they say, there’s various reasons
we don’t have a good brain theory. Some say we still don’t have enough data, we need more information,
there’s all these things we don’t know. Well, I just told you there’s data
coming out of your ears. We have so much information,
we don’t even know how to organize it. What good is more going to do? Maybe we’ll be lucky and discover
some magic thing, but I don’t think so. This is a symptom of the fact
that we just don’t have a theory. We don’t need more data,
we need a good theory. Another one is sometimes people say, “Brains are so complex,
it’ll take another 50 years.” I even think Chris said something
like this yesterday, something like, it’s one of the most complicated
things in the universe. That’s not true — you’re more
complicated than your brain. You’ve got a brain. And although the brain
looks very complicated, things look complicated
until you understand them. That’s always been the case. So we can say, my neocortex,
the part of the brain I’m interested in, has 30 billion cells. But, you know what?
It’s very, very regular. In fact, it looks like it’s the same thing
repeated over and over again. It’s not as complex as it looks.
That’s not the issue. Some people say,
brains can’t understand brains. Very Zen-like. Woo. (Laughter) You know, it sounds good, but why?
I mean, what’s the point? It’s just a bunch of cells.
You understand your liver. It’s got a lot of cells in it too, right? So, you know, I don’t think
there’s anything to that. And finally, some people say, “I don’t feel like a bunch
of cells — I’m conscious. I’ve got this experience,
I’m in the world. I can’t be just a bunch of cells.” Well, people used to believe
there was a life force to be living, and we now know
that’s really not true at all. And there’s really no evidence, other than that people just disbelieve
that cells can do what they do. So some people have fallen
into the pit of metaphysical dualism, some really smart people, too,
but we can reject all that. (Laughter) No, there’s something else, something really fundamental, and it is: another reason why we don’t have
a good brain theory is because we have an intuitive,
strongly held but incorrect assumption that has prevented us
from seeing the answer. There’s something we believe that just,
it’s obvious, but it’s wrong. Now, there’s a history of this in science
and before I tell you what it is, I’ll tell you about the history
of it in science. Look at other scientific revolutions — the solar system, that’s Copernicus, Darwin’s evolution,
and tectonic plates, that’s Wegener. They all have a lot in common
with brain science. First, they had a lot
of unexplained data. A lot of it. But it got more manageable
once they had a theory. The best minds were stumped —
really smart people. We’re not smarter now than they were then; it just turns out it’s really
hard to think of things, but once you’ve thought of them,
it’s easy to understand. My daughters understood
these three theories, in their basic framework, in kindergarten. It’s not that hard —
here’s the apple, here’s the orange, the Earth goes around, that kind of stuff. Another thing is the answer
was there all along, but we kind of ignored it
because of this obvious thing. It was an intuitive,
strongly held belief that was wrong. In the case of the solar system, the idea that the Earth is spinning, the surface is going
a thousand miles an hour, and it’s going through the solar system
at a million miles an hour — this is lunacy; we all know
the Earth isn’t moving. Do you feel like you’re moving
a thousand miles an hour? If you said Earth was spinning
around in space and was huge — they would lock you up,
that’s what they did back then. So it was intuitive and obvious.
Now, what about evolution? Evolution, same thing. We taught our kids the Bible says
God created all these species, cats are cats; dogs are dogs;
people are people; plants are plants; they don’t change. Noah put them on the ark
in that order, blah, blah. The fact is, if you believe in evolution,
we all have a common ancestor. We all have a common ancestor
with the plant in the lobby! This is what evolution tells us.
And it’s true. It’s kind of unbelievable. And the same thing about tectonic plates. All the mountains and the continents are kind of floating around
on top of the Earth. It doesn’t make any sense. So what is the intuitive,
but incorrect assumption, that’s kept us from understanding brains? I’ll tell you. It’ll seem obvious
that it’s correct. That’s the point. Then I’ll make an argument why
you’re incorrect on the other assumption. The intuitive but obvious thing is: somehow, intelligence
is defined by behavior; we’re intelligent
because of how we do things and how we behave intelligently. And I’m going to tell you that’s wrong. Intelligence is defined by prediction. I’m going to work you
through this in a few slides, and give you an example
of what this means. Here’s a system. Engineers and scientists
like to look at systems like this. They say, we have a thing in a box.
We have its inputs and outputs. The AI people said, the thing in the box
is a programmable computer, because it’s equivalent to a brain. We’ll feed it some inputs and get it
to do something, have some behavior. Alan Turing defined the Turing test,
which essentially says, we’ll know if something’s intelligent
if it behaves identical to a human — a behavioral metric
of what intelligence is that has stuck in our minds
for a long time. Reality, though —
I call it real intelligence. Real intelligence
is built on something else. We experience the world
through a sequence of patterns, and we store them, and we recall them. When we recall them,
we match them up against reality, and we’re making predictions all the time. It’s an internal metric;
there’s an internal metric about us, saying, do we understand the world,
am I making predictions, and so on. You’re all being intelligent now,
but you’re not doing anything. Maybe you’re scratching yourself,
but you’re not doing anything. But you’re being intelligent;
you’re understanding what I’m saying. Because you’re intelligent
and you speak English, you know the word at the end of this sentence. The word came to you;
you make these predictions all the time. What I’m saying is, the internal prediction
is the output in the neocortex, and somehow, prediction
leads to intelligent behavior. Here’s how that happens: Let’s start with a non-intelligent brain. I’ll argue a non-intelligent brain,
we’ll call it an old brain. And we’ll say it’s
a non-mammal, like a reptile, say, an alligator; we have an alligator. And the alligator has
some very sophisticated senses. It’s got good eyes and ears
and touch senses and so on, a mouth and a nose. It has very complex behavior. It can run and hide. It has fears
and emotions. It can eat you. It can attack.
It can do all kinds of stuff. But we don’t consider
the alligator very intelligent, not in a human sort of way. But it has all this complex
behavior already. Now in evolution, what happened? First thing that happened
in evolution with mammals is we started to develop a thing
called the neocortex. I’m going to represent the neocortex
by this box on top of the old brain. Neocortex means “new layer.”
It’s a new layer on top of your brain. It’s the wrinkly thing
on the top of your head that got wrinkly because it got shoved
in there and doesn’t fit. (Laughter) Literally, it’s about the size
of a table napkin and doesn’t fit, so it’s wrinkly. Now, look at how I’ve drawn this. The old brain is still there. You still have that alligator brain.
You do. It’s your emotional brain. It’s all those gut reactions you have. On top of it, we have this memory system
called the neocortex. And the memory system is sitting
over the sensory part of the brain. So as the sensory input
comes in and feeds from the old brain, it also goes up into the neocortex. And the neocortex is just memorizing. It’s sitting there saying, I’m going
to memorize all the things going on: where I’ve been, people I’ve seen,
things I’ve heard, and so on. And in the future, when it sees
something similar to that again, in a similar environment,
or the exact same environment, it’ll start playing it back:
“Oh, I’ve been here before,” and when you were here before,
this happened next. It allows you to predict the future. It literally feeds back
the signals into your brain; they’ll let you see
what’s going to happen next, will let you hear the word
“sentence” before I said it. And it’s this feeding
back into the old brain that will allow you to make
more intelligent decisions. This is the most important slide
of my talk, so I’ll dwell on it a little. And all the time you say,
“Oh, I can predict things,” so if you’re a rat and you go
through a maze, and you learn the maze, next time you’re in one,
you have the same behavior. But suddenly, you’re smarter;
you say, “I recognize this maze, I know which way to go; I’ve been here
before; I can envision the future.” That’s what it’s doing. This is true for all mammals — in humans, it got a lot worse. Humans actually developed
the front of the neocortex, called the anterior part of the neocortex. And nature did a little trick. It copied the posterior,
the back part, which is sensory, and put it in the front. Humans uniquely have
the same mechanism on the front, but we use it for motor control. So we’re now able to do very sophisticated
motor planning, things like that. I don’t have time to explain,
but to understand how a brain works, you have to understand how the first part
of the mammalian neocortex works, how it is we store patterns
and make predictions. Let me give you
a few examples of predictions. I already said the word “sentence.” In music, if you’ve heard a song before, when you hear it, the next note
pops into your head already — you anticipate it. With an album, at the end of a song,
the next song pops into your head. It happens all the time,
you make predictions. I have this thing called
the “altered door” thought experiment. It says, you have a door at home; when you’re here, I’m changing it — I’ve got a guy back at your house
right now, moving the door around, moving your doorknob over two inches. When you go home tonight, you’ll put
your hand out, reach for the doorknob, notice it’s in the wrong spot and go, “Whoa, something happened.” It may take a second,
but something happened. I can change your doorknob
in other ways — make it larger, smaller, change
its brass to silver, make it a lever, I can change the door;
put colors on, put windows in. I can change a thousand things
about your door and in the two seconds
you take to open it, you’ll notice something has changed. Now, the engineering approach,
the AI approach to this, is to build a door database
with all the door attributes. And as you go up to the door,
we check them off one at time: door, door, color … We don’t do that.
Your brain doesn’t do that. Your brain is making
constant predictions all the time about what will happen
in your environment. As I put my hand on this table,
I expect to feel it stop. When I walk, every step,
if I missed it by an eighth of an inch, I’ll know something has changed. You’re constantly making predictions
about your environment. I’ll talk about vision, briefly. This is a picture of a woman. When we look at people, our eyes saccade
over two to three times a second. We’re not aware of it,
but our eyes are always moving. When we look at a face, we typically
go from eye to eye to nose to mouth. When your eye moves from eye to eye, if there was something
else there like a nose, you’d see a nose where an eye
is supposed to be and go, “Oh, shit!” (Laughter) “There’s something wrong
about this person.” That’s because you’re making a prediction. It’s not like you just look over and say,
“What am I seeing? A nose? OK.” No, you have an expectation
of what you’re going to see. Every single moment. And finally, let’s think
about how we test intelligence. We test it by prediction:
What is the next word in this …? This is to this as this is to this.
What is the next number in this sentence? Here’s three visions of an object.
What’s the fourth one? That’s how we test it.
It’s all about prediction. So what is the recipe for brain theory? First of all, we have to have
the right framework. And the framework is a memory framework, not a computational or behavior framework, it’s a memory framework. How do you store and recall
these sequences of patterns? It’s spatiotemporal patterns. Then, if in that framework,
you take a bunch of theoreticians — biologists generally
are not good theoreticians. Not always, but generally, there’s not
a good history of theory in biology. I’ve found the best people
to work with are physicists, engineers and mathematicians, who tend to think algorithmically. Then they have to learn
the anatomy and the physiology. You have to make these theories
very realistic in anatomical terms. Anyone who tells you their theory
about how the brain works and doesn’t tell you exactly
how it’s working and how the wiring works — it’s not a theory. And that’s what we do
at the Redwood Neuroscience Institute. I’d love to tell you we’re making
fantastic progress in this thing, and I expect to be back on this stage
sometime in the not too distant future, to tell you about it. I’m really excited;
this is not going to take 50 years. What will brain theory look like? First of all, it’s going
to be about memory. Not like computer memory —
not at all like computer memory. It’s very different. It’s a memory of very
high-dimensional patterns, like the things that come from your eyes. It’s also memory of sequences: you cannot learn or recall anything
outside of a sequence. A song must be heard
in sequence over time, and you must play it back
in sequence over time. And these sequences
are auto-associatively recalled, so if I see something, I hear something,
it reminds me of it, and it plays back automatically. It’s an automatic playback. And prediction of future inputs
is the desired output. And as I said, the theory
must be biologically accurate, it must be testable
and you must be able to build it. If you don’t build it,
you don’t understand it. One more slide. What is this going to result in? Are we going to really build
intelligent machines? Absolutely. And it’s going to be
different than people think. No doubt that it’s going
to happen, in my mind. First of all, we’re going to build
this stuff out of silicon. The same techniques we use to build
silicon computer memories, we can use here. But they’re very different
types of memories. And we’ll attach
these memories to sensors, and the sensors will experience
real-live, real-world data, and learn about their environment. Now, it’s very unlikely the first things
you’ll see are like robots. Not that robots aren’t useful;
people can build robots. But the robotics part is the hardest part.
That’s old brain. That’s really hard. The new brain is easier
than the old brain. So first we’ll do things
that don’t require a lot of robotics. So you’re not going to see C-3PO. You’re going to see things
more like intelligent cars that really understand
what traffic is, what driving is and have learned that cars
with the blinkers on for half a minute probably aren’t going to turn. (Laughter) We can also do intelligent
security systems. Anytime we’re basically using our brain
but not doing a lot of mechanics — those are the things
that will happen first. But ultimately, the world’s the limit. I don’t know how this will turn out. I know a lot of people who invented
the microprocessor. And if you talk to them, they knew what they were doing
was really significant, but they didn’t really know
what was going to happen. They couldn’t anticipate
cell phones and the Internet and all this kind of stuff. They just knew like,
“We’re going to build calculators and traffic-light controllers. But it’s going to be big!” In the same way, brain science
and these memories are going to be a very
fundamental technology, and it will lead to unbelievable changes
in the next 100 years. And I’m most excited about
how we’re going to use them in science. So I think that’s all my time — I’m over, and I’m going to end my talk right there.