Jeff Hawkins is best known for bringing us the Palm Pilot, but he’s working on something that could be much, much bigger.
For the past several years, Hawkins has been studying how the human brain functions with the hope of replicating it in software. In 2004, he published a book about his findings. In 2012, Numenta, the company he founded to commercialize his work, finally showed itself to the world after roughly seven years operating in stealth mode.
I recently spoke with Hawkins to get his take on why his approach to artificial intelligence will ultimately overtake other approaches, including the white-hot field of deep learning. We also discussed how Numenta has survived some early business hiccups and how he plans to keep the lights on and the money flowing in.
An edited version of the interview follows. Hawkins kicks if off with a description of Numenta’s technology — which it calls hierarchical temporal memory — and how it came to be.
Derrick Harris: Please explain Numenta’s approach to brain-inspired artificial intelligence technology.
Jeff Hawkins: We’re going through a transition right now in the world of machine intelligence that’s similar to the transition from analog to digital computing back in the 1940s. Today, if you look in the world of machine learning and machine intelligence, you see varied types of things going on. There are different types of algorithms that people are using — specific and universal — and people debate which approach is better.
We’re very confident that by the end of the 2020s, we’re going to be settled on a dominant paradigm. It’s going to be quite different than the one we’re currently in today, where specific algorithms that excel at one task dominate. We believe it’s going to be based instead on the universal algorithms that work on many problems. They’re going to be memory-based, not mathematically based. They’re going to be based primarily on time-based patterns, and they’re going to be online learning paradigms.
[pullquote person=”” attribution=””]What I’ll say is the technology we’re developing is going to be huge. It’s a foundation for the next 60, 70, 100 years of computing.[/pullquote] Our belief in this comes really from studying the brain. This is what the neocortex does. The neocortex uses one framework for vision, audition, language, motor, planning, robotics, everything. Everything you do, writing, thoughts, planning conferences, it’s all the same thing. It’s one memory framework.
We’ve invented a term called hierarchical temporal memory, which describes the basic theory about what’s going on here. Very importantly, nothing in HTM is task-specific, just like in your brain. The way you see and the way you hear and the way you feel, the neural tissue that does that is exactly the same. You can actually swap them around, and you’ll still work.
That’s the key part of the whole bet here, that there are going to be, as we know in the brain, universal learning algorithms that may not be the best at everything, but they’re really good at everything.
What about the business side of things? I first wrote about Numenta in 2013 around some work you were doing with some energy industry customers. Then you pulled back, and then the new Grok app for Amazon Web Services came out. Is the technology different in any way since then?
Between what we were doing in the energy space and what we did with Grok, it’s pretty much the same technology. What we found in the energy space was that although we had all these companies that wanted work with us, none of them were ready to deploy.
We were basically doing energy predictions for something called demand response. The idea was that if you could predict future energy requirements, and future energy prices, you can adjust your consumption and save a lot of money. We did a really great job with this. All these companies were working with us, but they actually weren’t able to deploy it anywhere because they just didn’t have the infrastructure in place to actually act on our predictions.
It took us a while to realize that this was going to take a long time. We said, “Fine, we can’t wait around for that.” Then we said, “Let’s take a market space where we can build something today, and companies can deploy it today, and we don’t have anybody waiting around.”
You’ve compared your approach to deep learning and other approaches to AI that are really good at specific tasks, and talked about how the neocortex is using the same framework for vision and motor skills. Can Numenta’s technology actually handle computer vision tasks, or is it just a pattern recognition layer?
We can. There are two basic types of inference of pattern recognition that occurs in the brain. One is what we call sensory motor inference, which is how you recognize the world through movement. Vision actually is mostly a sensory motor inference problem. That is, your eyes are moving all the time. You’re not aware of it, but three to five times a second, you have completely different patterns coming into your eyes even though the world seems stable to you.
The brain uses those movements to build a model of the world. It’s not something that the brain is trying to get around. It’s not an inconvenience. It’s a part of how the brain sees. We use the same thing when you walk into a building. As you walk down a hallway going out doors and reach and touch things, most of the way you interact with the world is through your own behavior. It’s called sensory motor inference.
[pullquote person=”” attribution=””]I think what will happen is within the next few years, the killer apps will emerge, and I don’t think it you or I or anybody else would have anticipated them.[/pullquote]The other method is what we call high order inference. High order inference means that there are high order patterns. It’s a predictable sequence. It’s like a melody. You understand my speech, and your brain is constantly predicting what I’m going to say next, and what the difference sounds are going to be.
Grok uses high order inference. There’s no behavior in Grok, it just listens to the sounds of patterns coming off the servers.
Now, to do vision correctly, you need to do sensory motor inference. We understand that and we’re in the process of building it out now. That’s been a major research effort for us, starting in January. We think we can get to a vision system that is cortical like. It will work the same way the brain does. I have faith that it will be better than other approaches, but I can’t prove that yet. I do have a path to get there.
We’re currently working on that, but I can’t sit here today and say how our vision system performs compared to Google’s system or something like that.
While object recognition is useful and novel and very valuable in some ways, I can see a future where you definitely want to be able to recognize some of the higher-order stuff. Can you get to a point with Numenta where a system can actually make inferences beyond that an object is a thing?
Our basic approach is adhering to neuroscience principles so that we will get the properties that brains have. This idea that something is not just a cat or a dog, but what does it mean? What is it doing? What are the implications of this image? These come about because of the nature of what are called sparse distributed representations, and the way the brain forms in a hierarchy.
It’s a big bet that if you understand these cortical principles and you build the thing to work on those cortical principles, you will get the properties that human brains exhibit. I can’t prove that, but it seems pretty obvious to me that’s going to happen. We understand why it will happen now. It’s not just a guess.
The approach we’re taking is a very long-term approach. We’re trying to lay the foundation for the next 50 to 100 years of machine intelligence and machine learning. That takes a bit more patience, and it means that you spend more time building these foundational theories than going out and implementing stuff and reporting some results in some benchmark.
How much time do you think you have to make it work? How much runway do you have as a company?
We’re working on five different applications that are variations of the high order inference and prediction that I talked about. We did Grok, the real product. We’ve done a bunch of other things in detecting human behavior — changes in human behavior like if I’m trying to identify someone who becomes a rogue trader, or someone who starts stealing inside company information. We’re able to make predictions in tech anomalies and stock volumes.
[pullquote person=”” attribution=””]Intelligence is not dangerous. Self-replication is dangerous. A stupid self-replicating thing is sometimes more dangerous than a smart self-replicating thing, but either way self-replication is the thing we have to worry about.[/pullquote]We’ve done something in the geospatial tracking space, which is very cool. We can detect anomalies in movements of object and people, and we’ve also done some work with another company called Cortical.io doing natural language for improved search. This looks very promising. This is using the same basic code base. We haven’t even touched a line code, and you can apply it to all these different problems. We’ve already shown there’s a lot of diversity to this, and we’re starting to talk to people about licensing these different applications.
I should mention that when we announced Grok back in the beginning of the year, immediately we were approached by large companies that want to license it. We said, “Let’s make that our business strategy.” We’re revealing a series of applications, and we think ultimately people will want to license this stuff. There will be a series of more applications when we put in the sensory motor stuff, which is going to be a bit longer.
The problems we solve today are very different than what most people solve in deep learning. They basically can solve spatial pattern-recognition problems. They’re not really looking for time-based patterns. They don’t use prediction or anomaly detection. There’s not a lot of overlap between what they do and what we do.
It’s not like this is a zero-sum game, correct? Isn’t there room for your approach, and a deep learning approach and other approaches, to coexist?
I believe that they have to come together in the end. It’s like analog computing and digital computing coexisted for a few years, but in the end they didn’t coexist. The need for network effects is pushing us in a direction where in the end they will ultimately come together, or they will die.
I think deep learning is a great example. If you speak to Andrew Ng, he’ll say we have to add time deep learning networks. Deep learning networks are basically hierarchical networks, but they don’t have time. They don’t have a sense of prediction. They don’t have a sense of motor behavior.
If these fields don’t merge together, if people don’t make that effort, then there’s going to be orphans. I don’t believe it’s going to be like there will be lots of different techniques. I really believe it’s going to settle on a dominant paradigm.
We’ve been working with a large semiconductor manufacturer now for over a year that is very forward-thinking. It has a team of people developing hardware for our algorithms. It has chosen the HTM algorithms over anything else because it’s thinking 5, 10, 15, 20 years out and wants to pick the winning machine learning platform. Even if today it seems a little bit embryonic, it’s got legs to it.
I hear quantum computing referenced as this holy grail because it will let us do new things in machine learning. What does it mean for all the machine learning work being done on today’s systems if quantum computing becomes commercially viable?
I’m a brain guy. I’m a neuroscientist and a computer scientist, but if you look at the brain, it’s not magic. There’s no quantum weirdness going on there. It’s a very complex neural system. You have to understand why it looks the way it does, and then you can implement it in silicon.
At the moment, it looks like the most promising methods for implementing these things in hardware does not require any new device physics. They require new connectivity schemes. The big problem in building brains in silicon is connectivity, connecting memory in a way that it’s a huge memory system in a very large connectivity matrix. That’s the challenge in building artificial brains.
What do you see as the killer application for all this? Some of the stuff you’re doing right now with anomaly detection or pattern recognition, or computer vision? Maybe it’s superhuman AI agents.
History tells us something here. When a new technology paradigm or a new science paradigm or a new type of machine intelligence comes around, history tells you that the obvious applications are not the killer apps. The things that people thought, like what they were doing in the past, turned out not to be the killer apps. The killer apps tend to be surprising. No one anticipates them.
[pullquote person=”” attribution=””][I]f you look at the brain, it’s not magic. There’s no quantum weirdness going on there. … You have to understand why it looks the way it does, and then you can implement it in silicon.[/pullquote]When they first started building computers, of course, it was all about calculating numbers tables. They were using it to calculate ballistic trajectory tables. They were using it to calculate logarithm tables. The killer apps in computers initially turned out to be data processing in businesses. Nobody anticipated that in the 1940s. IBM stumbled upon that.
One of the things that we’ve done in Numenta is we don’t want to place a bet too strongly on any one of these things because we just don’t know. We’ll see how it goes, how many people like it, what’s going to happen with it, but we won’t bet the company on it. Something can generate good license revenue and be a good business without necessarily being the killer app.
I think what will happen is within the next few years, the killer apps will emerge, and I don’t think it you or I or anybody else would have anticipated them.
Is robotics the ultimate medium for these algorithms?
The principles of robotics have nothing to do with a physical embodiment. They have to do with an agent moving through a world. As you move, your sensory input changes. What you perceive changes because you’re moving through the world. Why does it have to be physical world? You can build a robot that works in a virtual world.
Web crawlers are very simple robots. They just follow links on the World Wide Web, that’s what they do. They’re like cockroaches that follow along, and just bump the wall, and they scour the web that way. You could build a robot system that’s a virtual system that basically moves through cyberspace. It learns to navigate in cyberspace, and learns to build a model of the world in the same way you and I build a model by walking through our towns and our houses.
This is what I find very interesting. I have another idea for what potentially could be a killer app in the same vein. Data itself has structure. If I look at a spreadsheet or some big database, it has structure to it. We might be able to apply our algorithms, our robotic algorithms, to essentially dig around in data to discover the structure in static data in the same way a human machine learning expert does.
I do think — just to hover on another topic that you mentioned — the whole idea of superhuman intelligence is something that people get wrong completely. We’re going to build machines that are far more intelligent than humans, but they’re not going to be human-like at all. They’re not going to be conversing with you, or threatening you, or anything like that.
Superhuman intelligence to me is basically just building a brain that’s really big and really fast, and has some very interesting sensory organs, but it doesn’t have to be human-like. I don’t think we’re going to build human-like things. There’s no point in that.
Your example of a robot that would move through cyberspace, that’s actually a very viable sort of superintelligent thing.
It’s not a threat to humanity.
Well, it could be. A very smart bot or virus roaming through cyberspace could wreak a lot of havoc.
You said the key word there. The word is virus. The thing that we have to be concerned about from a safety point of view is anything that replicates. Intelligence is not dangerous. Self-replication is dangerous. A stupid self-replicating thing is sometimes more dangerous than a smart self-replicating thing, but either way self-replication is the thing we have to worry about.
There have been some notable personnel and business changes at Numenta. At this point, is the company in a good place to go forward and really capitalize on the technology?
A year from now would we change something? I don’t know. Possibly. But I haven’t felt this good about Numenta for a long time. One, from the science point of view, the technology point of view, we’re making great progress. We’ve made big progress just the beginning of this year on the sensory motor stuff. We’ve proven the other technology works well.
We’ve also had independent validation that the technology is valuable. We have people who have said, “Yup, this is really cool. We want to buy this. It’s worth a lot to us.”
[pullquote person=”” attribution=””]We’re going through a transition right now in the world of machine intelligence that’s similar to the transition from analog to digital computing back in the 1940s.[/pullquote]We’ve also had independent verification that the concepts are being adopted. I mentioned that there are some large companies that are investing significant dollars in these algorithms. There’s a program at DARPA that is based on HTMs. There’s another one that’s very similar to it in IARPA, which is the intelligence community. These people are heavily influenced by our work.
Our open source community is doing very well. That’s been around for about a year. I’m feeling that we’ve turned a corner here, so I’m a little more optimistic.
I think also our business strategy and our system strategy is great. We may create a hit product along the way, but we’re not counting on that. If that happens, great. We’re going to building products until we get better at it, and eventually something will click, or someone will buy the company or something like that.
Given the technology we’re talking about, how big can a company like Numenta be if it actually hits a homerun? Or will Google acquire it first and that will be that?
It’s hard to say. Again, I like history. If you go back when people first started building computers in the late 1940s, there were a whole bunch of companies formed. They were fighting each other. There were patent wars going on at this time. Almost all those companies killed each other. IBM snuck up on the outside and made a business out of it.
What I’ll say is the technology we’re developing is going to be huge. It’s a foundation for the next 60, 70, 100 years of computing. It’s not replacing computing, but it’s as a big as computing. We’re pioneers, and we’re laying a foundation which will exist and survive.
As a business, it’s very tricky in the beginning of something like this to pick your bets and stay viable. How big Numenta will be, I have no idea. Our approach is to stay small as long as possible because that gives us flexibility. That gives us the ability to change this. As soon as you start becoming big, then you’ve chosen a path. Then you become obsolete in a few years.
All I can say is that I believe these machine learning concepts that are coming from the brain are going to be the central ones that are used in machine learning and machine intelligence. That’s going to be a monstrous industry, multiple industries going forward for decades. We’re going to see how to keep ourselves relevant and valuable as long as we can.