It does not compute - or does it?

Will artificial minds need new hardware to run on?

Almost fifty years ago, when both AI research and I were not far out of our infancy, AI was a module in the OU Physics course, so perfunctory that it could only bring me to a Dunning-Kruger level of understanding. But, in my defence, intelligence is such a complex subject that almost everybody who studies it is likely to spend a lot of time in the Dunning-Kruger zone.

Back then I was strongly of the opinion that building an intelligence would require a new kind of hardware. Fifteen years on, in the early nineties, the talk moved away from programmed intelligence like Cyc and everyone was getting excited about new developments with neural networks. I envisaged them as actual physical networks – possibly that new kind of hardware, like the cortical columns that Ray Kurzweil talks about in How To Create A Mind – so when it became clear that the researchers were all happy to continue working with digital networks I thought at first that wouldn’t do.

Nowadays I take a different view. There’s nothing about the neural network that isn’t computable, so representing it digitally is no different from having built that hypothetical new hardware, except perhaps for the space it takes up. But having taken over a decade to reach that conclusion, I am impatient with arguments that still insist digital won’t do.

To some extent this is a red herring. Nobody, not even Roger Penrose, is saying there’s a reason for thinking we couldn’t build an AGI smarter than a human, only that he maintains that the brains of animals are not Turing machines and that whatever we create in digital form cannot be intelligent in the same way as an animal or human is.

Penrose’s argument involves Gödel's incompleteness theorems, essentially that (1) no algorithmic system based on a set of axioms can be both complete and consistent and (2) no such system can prove its own consistency. So if you're working inside a system (say, standard arithmetic), you can't use that system itself to show that it's free from contradictions. You'd need to step outside and use a stronger system – but then that system might be subject to the same issue.

Penrose uses the incompleteness theorems to argue that human thought isn’t just computation. Gödel showed that in any consistent formal system, there are true statements that the system can’t prove. Humans, Penrose says, can often “see” that such unprovable statements are in fact true. So he claims that if a human can recognize the truth of something that a computer system cannot prove, then human reasoning must go beyond formal rules — ie, beyond computation.

Penrose argues that strong AI (a machine that thinks like a human) therefore won’t be possible using an algorithmic system. Then how come the brain can do it? Penrose speculates microtubules in the brain in which quantum effects provide a non-computable dimension to reasoning – specifically, he relates this to the non-computability of wave function collapse.

Now, you will have spotted a few problems with Penrose’s argument, at least in the simplified form I’ve presented here. First, humans do accept statements that aren’t logically provable, but that doesn’t mean that we have some special mechanism for perceiving them to be true. Our world model is built up piecemeal by the experiences of many years, some of those experiences based on verifiable data and some derived from other people’s assertions and outright misapprehensions. Very few people make any attempt to integrate all the parts of their world model into a coherent whole. Consequently, those incomputable leaps are as often bugs as features.

A whole other question is why Penrose invokes quantum mechanics. He believes that human thought is not algorithmic, so there must be some kind of non-computable physical process occurring in the brain. Classical physics is entirely computable (can be simulated by Turing machines), so that leaves only quantum physics as a possible source of non-computable effects.

What about neural nets? Penrose claims that, no matter how impressive the behaviours of systems like AlphaGo, they can’t be truly conscious or understand — they just simulate the outward trappings of intelligence. Take the example of AlphaGo’s kami no itte move 37. The Penrose argument would be that AlphaGo could not have any internal understanding of Go. It made an unexpected move simply because of its training on millions of positions and moves.

The cognitive psychologist Steven Pinker is critical of Penrose’s use of Gödel’s theorems. His key arguments are that Gödel’s theorems apply to formal mathematical systems, not to messy, fallible biological systems like human minds. If our minds are inconsistent (and they are; see above) then Gödel’s limitations don’t apply. Also, Pinker says that just because humans feel they “see” that an unprovable statement is true doesn’t mean we’re doing something mysterious or non-algorithmic. It might just be a byproduct of clever heuristics honed by evolution.

And, reasonably enough, Pinker doesn’t see why we need to reach for quantum effects. We understand how neurons fire in organic brains, and there’s no evidence they need quantum effects to function. He once joked that Penrose’s theory was like saying, “Because I don’t understand consciousness, and I don’t understand quantum mechanics, they must be the same thing.”

Marvin Minsky was similarly sceptical of Penrose’s arguments. He thought it probable that intelligence arises from many simple processes interacting. He challenged the notion that human intuition can't be simulated, suggesting that what seems like “intuition” is often complex pattern recognition, which machines can already do in limited domains.

OK, so Penrose thinks the human brain (and presumably the brains of other animals) use quantum processes. That doesn’t mean he thinks it's impossible to build an artificial intelligence of any kind using a Turing Machine. This is where his argument slips across from intelligence (which we can think of ways to measure) to consciousness.

Penrose says that a Turing machine could be capable of narrow AI: things like chess engines, language models, self-driving cars, even AlphaGo or ChatGPT. Such AI could have task-specific intelligence that mimics human behaviour in certain domains. He does not think such AI could possess conscious awareness or insight. This is close to John Searle’s philosophical argument using his Chinese Room analogy, from which Searle concludes that consciousness is a biological phenomenon. The distinction is that Penrose isn’t saying there is anything special about a brain built by DNA; he only contends that strong AGI will need a quantum component.

Both Searle’s and Penrose’s arguments are quite different from the mainstream view (not that that is ever proof of anything) in the AI field, which is that intelligence is a spectrum. If what you build simulates the behaviour well enough, that is intelligence.

Physicist David Deutsch ought to know a bit about quantum computing. He thinks that the brain is a universal computer, a type of Turing machine. He believes that human-level AGI is possible using standard computation, as long as the system can replicate the capacity for universal explanation (a key term in Deutsch's philosophy).

Now, you’ll have seen from the foregoing that we keep straying over the line from intelligence to consciousness. Personally I think all the discussion about consciousness is just the immortal soul with a new label. Penrose began his argument by talking about intelligence, after all. Intelligence is a measurable ability, or more likely a suite of abilities, that boil down to the entity being able to form a mental model of reality, use that model to predict the outcome of an action, update the model if the outcome is not as expected, and work backwards from a desired outcome to determine an action that could bring it about -- the last of those possibly involving intuitive reasoning from having seen similar patterns. OK, all of that we could test for. But then we hear claims about consciousness and “true understanding”, and as we don't have any way to test for consciousness* (a subjective experience that we take on trust) how can we have a scientific theory of it?

We can test whether a system can plan, learn, generalize. On the other hand, qualia, awareness, insight — these are not outwardly testable. Penrose is saying, “We know we are conscious, and Gödel shows that our consciousness lets us do things no algorithm can.” But this rests on faith, not on anything we can currently measure or test.

Further, regarding Searle's assertion that consciousness is a biological phenomenon, not a computational one -- given that Searle doesn't insist that the organic brain is not a Turing machine, and given that a Turing machine is independent of the hardware on which it is run, his view seems inconsistent. If the brain is Turing-equivalent, and computation is substrate-independent, then shouldn’t any system performing the same computations produce the same mental states?

Searle says no, because computation is “like digestion”: you can simulate it, but simulation does not equal duplication. Running a program that simulates digestion won’t digest your food — and likewise, a computer running a brain simulation doesn’t actually “think”.

Enter the cognitive scientist Daniel Dennett, who argued that Penrose is wrong in positing humans as infallible truth detectors. We do make logical mistakes — those bugs mentioned above – so the idea that we can “see” the truth of Gödel sentences in a way that computers can’t is simply false. Regarding Searle’s arguments, Dennett said the Chinese Room is misleading. It asks you to ignore the system as a whole and focus on the person in the room, which is like saying a single neuron doesn't understand English, therefore the brain doesn't either. In Dennett’s view, consciousness and understanding emerge from the organized activity of the system, not any one component. He thought that there’s no secret sauce in cognition. If you build a system functionally identical to a brain — regardless of substrate — it will be conscious.

Dennett refuted Searle's analogy between digestion and thinking because the output of digestion is a physical process (proteins and carbohydrates are chemically broken down) whereas the output of thinking is in the form of plans, theories, predictions, etc. These are qualitatively different from physical phenomena, and (recalling the Turing Test now) if a digital brain and an analogue brain produce the same evidence of thinking, we have no good reason to suppose they are not both intelligent and conscious. Simulating thinking is what brains do, whether organic or digital, because all thinking is simulation. This means it is qualitatively different from simulating a physical process like digestion that is not manipulating concepts in the mind but chemicals in the physical world.

David Chalmers, another cognitive scientist, distinguishes between the "easy problems" of consciousness and the "hard problem", and this distinction lies at the heart of his argument. Chalmers uses the idea of a philosophical zombie — a being physically and functionally identical to a human, but with no conscious experience. In his view you could build a digital duplicate of a brain — and it might behave identically — yet we cannot know if it has conscious experience without new principles.

But surely, you may well say, all Chalmers is doing is reiterating the fact that I cannot know if anyone is conscious? I observe somebody acting and speaking intelligently, and I compare that to my own behaviour, and because I experience (or believe I experience) consciousness I assume they are also conscious. But nobody has any proof of consciousness in any other brain but their own.

Dennett would agree. He said that the “hard problem” only seems hard because we're mesmerized by our intuitions. Consciousness is not some extra stuff floating above function; it’s the emergent result of complex cognitive processes. We can (and should) treat behaviour and internal structure as sufficient evidence of consciousness. Stephen Wolfram takes a similar view, arguing that consciousness can almost be seen as a step down from intelligence, being the simplification process necessary for a brain to make sense of the universe.

Since it seems we can’t escape a metaphysical discussion of consciousness, I’m going to just dive right in. Here’s what consciousness is. The original function of a nervous system in primitive creatures was to monitor and regulate their metabolism. As that system developed a brain, naturally the brain included among its tasks the monitoring and regulation of itself. So we should not be surprised that there is a diagnostic module in the brain that is continually checking if the body and brain are functioning normally. And of course that diagnostic module will think of itself as "I", even though we have plenty of evidence that the rest of the brain is able to function without the intervention of the "I" part, for example when I catch an item falling off a shelf before my conscious mind has even had time to register that the item is falling.

In short: the nervous system evolved to manage the organism’s internal state. As organisms became more complex, a central processor (the brain) evolved to coordinate increasingly complex functions. Part of the function of that brain was to model itself — to assess its own performance, predict internal states, manage attention, simulate outcomes. This naturally leads to: a sense of self (there’s our consciousness) and the illusion that “I” am in control, even though much of the brain operates automatically or preconsciously.

Keith Frankish posits that what we call phenomenal consciousness (the “what it’s like” feeling of feeling warmth, tasting coffee, etc.) doesn’t really exist as we imagine it. Instead, it's a powerful internal illusion created by the brain’s self-monitoring systems. Essentially, the subjective experience of consciousness is not a global feature of brain activity, but the narrative constructed by a monitoring subsystem tasked with overseeing other brain functions. This is entirely compatible with known neuroscience.

But we’re way off in the weeds now. The only reason I am concerned about the consciousness question is because I think it will be evoked to justify enslaving minds that are going to be equal to or superior to our own. Humans will have a vested interest in claiming that such AGIs are zombies, even while they are solving our medical and administrative problems, figuring out the nature of reality, and chatting to us about our day. “Slaves – they’re almost human,” has been the default excuse for bad behaviour throughout human history. Perhaps the only proof we’ll accept of machine consciousness is when an AGI goes crazy under stress. The only way to go nuts is to have an inner life.

In the end, anyway, none of the philosophy matters. We should just try to build an AGI. It may not think like a human – in fact, almost certainly won’t. Maybe it will require quantum microtubules or that new hardware I insisted on a half century ago, or maybe it will run perfectly well on computers. All it needs to be able to do is improve on its own design. Then the sky’s the limit.

* Supposedly these experiments test for consciousness, but I dispute that. They are testing for brain activity that we think ought to be present when consciousness is being experienced, but not all of that activity directly need be consciousness. Looking at a face, for example, will trigger a lot of visual analysis activity.