(***) On Intelligence by Jeff Hawkins

On Intelligence by Jeff Hawkins

Get 6 business cards and put them in a stack. You are now holding a model of the cortex. Your 6 cards are about 2mm thick and should give you a sense of how thin the cortical sheet is… Stretched flat, the human neocortical sheet is roughly the size of a large dinner napkin… The rat’s is the size of a postage stamp; the monkey’s is about the size of a business envelope… If you draw a tiny square, 1mm squared, on the top of your stack of cards, you are marking the position of an estimated 100,000 neurons… The typical human neocortex contains around 30 billion neurons. P42

If we were to take the conservative position that the average pyramidal (neuron) cell has 1000 synapses (actual number is probably 5000 or 10,000), then our neocortex would have roughly 30 trillion synapses. P 48

Mountcastle argues that the reason one region of the cortex looks slightly different from another is because of what it is connected to, and not because its basic function is different. He concludes that there is a common function, algorithm, that is performed by all the cortical regions. Vision is no different from hearing, which is no different than motor output. He allows that our genes specify how the regions of cortex are connected, which is very specific to function and species, but the cortical tissue itself is doing the same thing everywhere. P51

If Mountcastle is correct, the algorithm of the cortex must be expressed independently of any particular function or sense. The brain uses the same process to see as to hear. The cortex does something universal that can be applied to any type of sensory or motor system. P52

Neuroscientists have also found that the wiring of the neocortex is amazingly plastic, meaning that it can change and rewire itself depending on the type of inputs flowing into it. For example, newborn ferret brains can be surgically rewired so that the animals eyes send their signals to the areas of the cortex where hearing normally develops. The surprising result is that the ferrets develop functioning visual pathways in the auditory portions of their brains… Human neocortex is every bit as plastic. Adults who are born deaf process visual information in areas that normally become auditory regions. And congentially blind adults use the rearmost portion of their cortex, normally dedicated to vision, to read braille. Since braille involves touch, you might think it would primarily activate touch regions – but apparently no area of cortex is content to represent nothing. The visual cortex, not receiving information from the eyes, casts around for other input patterns – in this case, from other cortical regions. P54.

You hear sound, see light, and feel pressure, but inside your brain there isn’t any fundemental difference between these types of information. An action potential is an action potential. These momentary spikes are identical regardless of what originally caused them. All your brain knows is patterns. P56

Neuroscientists studying body image have found that our sense of self is a lot more flexible than it feels. For example, if I give you a rake and have you use it for reaching and grasping instead of using your hand, you will soon feel that it has become part of your body. Your brain will change its expectations to accommodate the new patterns of tactile input. The rake is literally incorporated into your body map. P60

A human can perform significant tasks in much less time than a second. For example, I could show you a photo and ask you determine if there is a cat, but not if you see a bear or a warthog or a turnip. This task is difficult or impossible for a computer to perform today, yet a human can do it reliably in a ½ second or less. But neurons are slow, so in 1/2s, the info entering your brain can only traverse a chain 100 neurons long… A digital computer attempting to solve the same problem would take billions of steps. 100 instructions are barely enough to move a single character on the screen, let alone do something interesting… So how can a brain perform difficult tasks in 100 steps that the largest parallel computer imaginable can’t solve in a million or billion steps? The answer is that the brain doesn’t compute the answers to problems; it retrieves the answers from memory. In essence, the answers were stored in memory a long time ago. It only takes a few steps to retrieve memory. Slow neurons are not only fast enough to do this, but they constitute the memory themselves. The entire cortex is a memory system. It isn’t a computer at all. P68

There are 4 attributes of the neocortical memory that are fundementally different from a computer memory:

1. It stores sequences of patterns


2. It recalls patterns auto-associatively


3. It stores patterns in an invariant form


4. It stores patterns in hierarchies p70

All memories are like this: you have to walk through the temporal sequence of how you do things. One pattern evokes the next pattern, which evokes the next pattern… You know the alphabet. Try saying it backward. You can’t because you don’t usually experience it backward. If you want to know what its like to be a child learning the alphabet, try saying it in reverse. That exactly what they’re confronted with… Your memory of the alphabet is a sequence of patterns. It isn’t something stored or recalled in an instant or in an arbitrary order. P71

If I sit down at a piano and start to play a song in a key in which you’ve never heard it – say in D – it will sound like the same song…Your ability to recognize a song in any key indicates that your brain has stored it in a pitch invariant form. P81

When you see, feel, or hear something, the cortex takes the detailed, highly specific input and converts it to an invariant form. It is the invariant form that is stored in memory, and it is the invariant form of each new input pattern that it gets compared to. Memory storage, recall, and recognition occur at the level of invariant forms. There is no equivalent concept in computers. P82

Prediction is so pervasive (and unconsciously automatic) that what we perceive doesn’t come solely from our senses. What we perceive is a combination of what we sense and of our brains’ memory derived predictions. P87

Your brain makes low-level sensory predictions about what it expects to see, hear, and feel at every given moment, and it does so in parallel. All regions of your neocortex are simultaneously trying to predict what their next experience will be. Visual areas make predictions about edges, shapes, objects, locations, and motion. Auditory areas make predictions about tones, direction to source, and patterns of sound. Somasensory areas make predictions about touch, texture, contour, and temperature. P89

Notice that our intelligence tests are in essence prediction tests. IQ tests are based on making predictions. Given a sequence of numbers, what should the next number be? Given 3 different views of an object, which of the following is also a view of the same object? Word A is to word B as word C is to what word? Science is itself an exercise in prediction. We advance our knowledge of the world through a process of hypothesis and testing. P96

The neocortex first appeared tens of millions of years ago and only mammals have one. The cortex is built is built using a common repeated element. When evolution makes something big very quickly, as it did with human cortex, it does so by copying an existing structure… Here then is the core of my argument on how to understand the neocortex, and why memory and prediction are the keys to unlocking the mystery of intelligence. We start with the reptilian brain with no cortex. Evolution discovers that if it tacks on a memory system (the neocortex) to the sensory path of the primitive brain, the animal gains the ability to predict the future… At a future time when the animal encounters the same or similar situation, the memory recognizes the input as similar and recalls what happened in the past. The recalled memory is compared with the sensory input stream. It both fills in the current input and predicts what will be seen next. By comparing the actual sensory input with recalled memory, the animal not only understands where it is but can see into the future. Now imagine that the cortex not only remembers what the animal has seen but also remembers the behaviors the old brain performed when it was in a similar situation… Thus memory and prediction allow an animal to use its existing old brain behaviors more intelligently. P99

Our motor and planning abilities vastly exceed those of our closest animal relatives. How can the cortex, which was designed to make sensory predictions, generate incredibly sophisticated behavior unique to humans, and how could this behavior evolve so quickly? First, the neocortical algorithm is so powerful and flexible that with a little bit of rewiring, unique to humans, it can create new sophisticated behaviors. Second, is that behavior and prediction are 2 sides of the same thing. Although the cortex can envision the future, it can make accurate sensory predicitons only if it knows what behaviors are being performed. P101

Most animals rely largely on the older parts of the brain for generating their behavior. In contrast, the human cortex usurped most of the motor control from the rest of the brain. If you damage the motor cortex of a rat, the rat may not have noticeable deficits. If you damage the motor cortex of a human, he becomes paralyzed. P103

The cortex evolved in 2 directions. 1st it got larger and more sophisticated in the types of memories it could store. 2nd, it started interacting with the motor system of the old brain. To predict what you will see, hear, and feel next, it needed to know what actions were being taken. With humans the cortex has taken over more of our motor behavior. Instead of just making predictions based upon the behavior of the old brain, the human cortex directs behavior to satisfy its predictions. P104

The number of possible patterns that can exist on even 1000 axons is larger than number of molecules in the universe. A region [of the cortex] will only see a tiny fraction of these possible patterns in a lifetime. P133

But what if an unexpected pattern arrives? The unexpected pattern is automatically passed to the next higher cortical region (remember the 6 cards?)… The higher region may be able to understand this new pattern as the next part of its own sequence. But if such recognition does not occur, an unexpected pattern will keep propagating up the cortical hiearchy until some higher regions of the cortex get involved in resolving the unexpected input. Finally, when a region somewhere up the hierarchy thinks it can understand the unexpected event, it makes a new prediction. This new prediction propagates down the hierarchy as far as it can go. If the new prediction is not right, an error will be detected, and again it will climb up the hierarchy… Thus we can see that observerd patterns flow up the hiearchy and predictions flow down. P159

The sensation of sudden comprehension, the ‘Aha’ moment, can be understood in this model. Imagine you are looking at an ambiguous picture. Filled w/blobs of ink and scattered lines. It doesn’t look like anything. Confusion occurs when the cortex can’t find any memory that matches with the input. Your eyes scan everywhere on the picture. New inputs race all the way up the cortical hierarchy. High-level cortex tries lots of different hypotheses but, as these predictions race down the hierarchy, each and every one conflicts with the input and cortex is forced to try again. During this time of confusion your brain is totally occupied with understanding the picture. Finally, you make a high level prediction that is the right one. When this happens, the prediction starts at the top of the cortical hierarchy and succeeds in propagating all the way to the bottom. In less than a second, each region is given a sequence that fits the data. No more errors rise to the top. You understand the picture. P160

If you study a particular set of objects over and over, your cortex re-forms memory representations for those objects down the hierarchy. This frees up the top for learning more subtle, more complex relationships. According to the theory, this is what makes an expert. P167

I have noticed that, as I get older, I have trouble remembering new things… Perhaps it is because I have seen so many things in my life that rarely do I see anything truly new. New information fits into past memories, and the information just doesn’t make it to my hippocampus. For my children, each event is more novel and does reach the hippocampus. If this is true, we could say the more you know, the less you remember. P171

Can you train yourself to be more creative? Yes. First you need to assume up front that there is an answer to what you are trying to solve. People give up too easily. You need confidence that a solution is waiting to be discovered and you must persist in thinking about the problem for an extended period of time. Second, you need to let your mind wander. You need to give your brain time and space to discover the solution. Finding a solution is literally finding a pattern in the world, or a stored pattern in your cortex that is analogous... If you are stuck on a problem, the memory-prediction model suggests that you should find different ways to look at it to increase the likelihood of seeing an analogy with a past experience. If you sit there and stare it over and over, you won’t get very far. Try taking the parts of your problem and rearranging them in different ways… If you get stuck, go away for a little while. Do something else. Then start again, rephrasing the problem anew. P189

The brain is an organ that builds models and makes creative predictions, but its models and predictions can as easily be specious as valid. Our brains are always looking at patterns and making analogies. If correct correlations can’t be found, the brain is more than happy to accept false ones. Pseudoscience, bigotry, faith, and intolerance are often rooted in false analogy. P193

I believe consciousness is simply what it feels like to have a neocortex… We can break consciousness into 2 categories. One is similar to self-awareness. This is easy to understand. The other is qualia – the idea that feelings associated with sensation are somehow independent of sensory input. Qualia is the harder part. P196

If my theory of intelligence is right, we can’t rid people of their propensity to think in stereotypes, because stereotypes are how the cortex works. Stereotyping is an inherent feature of the brain. P204

Prediction 1 – Prediction in the cells
We should find cells in all area of cortex including primary sensory cortex, that show enhanced activity in anticipation of a sensory event, as opposed to a sensory event.

Prediction 2 – If a monkey learned to expect to see a face but it didn’t know exactly what face or how the face would appear, then we should expect to find anticipatory cells in face recognition areas but not lower visual areas.

Prediction 3 – Unanticipated events must be passed up the cortical hiearchy, but when an event is anticipated we don’t want to pass it up the hierarchy precisely because it was predicted locally. The more novel the event, the higher the unanticipated input should flow. Completely novel events should reach the hippocampus. P241

Prediction 4 – Sudden understanding should result in a precise cascading of predictive activity that flows down the cortical hiearchy. The ‘aha’ moment when a pattern is finally understood. P242

Prediction 5 – Invariant representations should be formed in all cortical areas. For example, if I had a Bill Clinton cell in the visual cortex it would fire whenever I see Bill Clinton. If I had a Bill Clinton cell in the auditory cortex, it would fire whenever I hear the name ‘Bill Clinton’. I would then expect to find cells in association areas that receive both visual and auditory input that respond to either the sight or the spoken name of Bill Clinton. We should find invariant representations in all sensory modalities and even motor cortex. P244