(2026-01-15) Hoel Proving, literally, that ChatGPT Isn't Conscious

Erik Hoel: Proving (literally) that ChatGPT (LLM) isn't conscious.

Imagine we could prove that there is nothing it is like to be ChatGPT. Or any other Large Language Model (LLM). That they have no experiences associated with the text they produce

You may already believe this, but a proof would mean that a lot of people who think otherwise, including some major corporations, have been playing make believe. Just as a child easily grants consciousness to a doll, humans are predisposed to grant consciousness easily.

*However, without a proof, the current state of LLM consciousness discourse is closer to “Well, that’s just like, your opinion, man.”

This is because there is no scientific consensus around exactly how consciousness works*

although, at least, those in the field do mostly share a common definition of what we seek to understand. ("Consciousness is a great mystery. Its definition isn't.")

There are currently hundreds of scientific theories of consciousness trying to explain how the brain (or other systems, like AIs) generates subjective and private states of experience. I got my PhD in neuroscience helping develop one such theory of consciousness: Integrated Information Theory, working under Giulio Tononi, its creator. And I’ve studied consciousness all my life. But which theory out of these hundreds is correct? Who knows! Honestly? Probably none of them.

So, how would it be possible to rule out LLM consciousness altogether?

In a new paper, now up on arXiv, I prove that no non-trivial theory of consciousness could exist that grants consciousness to LLMs.

Essentially, meta-theoretic reasoning allows us to make statements about all possible theories of consciousness, and so lets us jump to the end of the debate: the conclusion of LLM non-consciousness.

What is uniquely powerful about this proof is that it requires you to believe nothing specific about consciousness other than a scientific theory of consciousness should be falsifiable and non-trivial. If you believe those things, you should deny LLM consciousness.

Before the details, I think it is helpful to say what this proof is not.

It is not arguing about probabilities. LLMs are not conscious.

It is not applying some theory of consciousness I happen to favor and asking you to believe its results.

It is not assuming that biological brains are special or magical.

It does not rule out all “artificial consciousness” in theory.

How the proof works

There’s no substitute for reading the actual paper.

But, I’ll try to give a gist of how the disproof works here via various paradoxes and examples.

First, you can think of testing theories of consciousness as having two parts: there are the predictions a theory makes about consciousness (which are things like “given data about its internal workings, what is the system conscious of?”) and then there are the inferences from the experimenter (which are things like “the system is reporting it saw the color red.”)

The structure of the disproof of LLM consciousness is based around the idea of substitutions within this formal framework, which means swapping between systems while keeping identical input/output

And if somehow predictions didn’t change following substitutions, that’d be a problem too, since it would mean that you wouldn’t need any details about the system implementing f for your theory… which would mean your theory is trivial!

Continual learning stands out

One marker of a good research program is if it contains new information. I was quite surprised when I realized the link to continual learning.

learning radically complicates input/output equivalence

Importantly, in the paper I show how this grounding of a theory of consciousness in learning must happen all the time; otherwise, substitutions become available, and all the problems of falsifiability and triviality rear their ugly heads for a theory.

Thus, real true continual learning (as in, literally happening with every experience) is now a priority target for falsifiable and non-trivial theories of consciousness.

And… this would make a lot of sense?

LLMs know so much, and are good at tests. They are intelligent (at least by any colloquial meaning of the word) while a human baby is not. But a human baby is learning all the time, and consciousness might be much more linked to the process of learning than its endpoint of intelligence.

Finally, a progressive research program for consciousness!

It is no secret that I’d become bearish about consciousness over the last decade.

I’m no longer so. In science the important thing to do is find the right thread, and then be relentless pulling on it. I think examining in great detail the formal requirements a theory of consciousness needs to meet is a very good thread. It’s like drawing the negative space around consciousness, ruling out the vast majority of existing theories, and ruling in what actually works.

Right now, the field is in a bad way. Lots of theories. Lots of opinions. Little to no progress.

Has a single theory of consciousness been clearly ruled out empirically? If no, what precisely are we doing here?

Another direction is needed. By creating a large formal apparatus around ensuring theories of consciousness are falsifiable and non-trivial (and grinding away most existing theories in the process) I think it’s possible to make serious headway on the problem of consciousness

It will lead to more outcomes like this disproof of LLM consciousness, expanding the classes of what we know to be non-conscious systems (which is incredibly ethically important, by the way!),

I’ve started a nonprofit research institute to do this: Bicameral Labs.