Artificial intelligence is not conscious – Ted Chiang

Totally understandable; I don't think we can fully understand the human brain, using the human brain. We can understand its principles (firings and chemistry, structure and specialized areas, etc) but otherwise it's a capacity problem.

And while I can't fully understand myself, let alone another person, I definitely enjoy talking with people and sharing thoughts that I realize I wouldn't have had on my own.

Humans appear to intelligenty communicate, however these are just cleverly disguised sound patterns produced by the brain that happen to increase the likelihood of food going into their mouths, and various similar reward attracting mechanisms that make survival outcomes more likely. So human intelligence could be reduced to something like "fancy food-attracting algorithms" using the same fallacy.

I'm kind of on the fence on the subject of whether LLMs could be compared to the complexity of the human brain, myself.

Personally, I'm partial to the higher-order theory of consciousness which postulates that consciousness constitutes patterns of thought that arise in response to first-order mental states. So, an external stimulus produces a pattern within the neural network which represents a sensation, and then if a pattern arises in response to that pattern, that is an experience of that sensation.

Given this framework we could ask whether LLMs experience higher order patterns in response to external stimulus. We would have a clear question to ask which is whether the system can observe itself.

https://www.goodreads.com/en/book/show/217432753-the-ai-con

which describes LLMs as "souped-up autocomplete", complex statistics that cannot truly understand anything. A more recent example is this paper:

https://zenodo.org/records/20071869

which says,

> [LLMs], as turbo-charged statistical models (recall their formal relation to logistic regression) can only but provide correlations.

And, of course, the Stochastic Parrot paper is the classic example in this area. It is from 5 years ago, but "LLMs only do statistics / can't understand" is very much alive and active among academics, even if it is a minority position.

My point is that the fact that AI can reproduce convincingly human sentence continuation does not imply that the AI has no choice but ending up using a mechanism that "understand" rather than just have learned data patterns that are very effective to fake human sentence continuation but are meaningless in term of understanding the concepts.

And I think that if indeed the only way for AI to reproduce convincingly human sentence continuation would be to end up in a configuration that uses the "understand" mechanism to do so, the behaviour of the first LLM would not show that they are so good at sounding human and yet so bad at failing basic "understanding" tests.

You are basically arguing for a functional account of consciousness, but things like this have been debated for literally decades/centuries in philosophy.

I am far from convinced that the training and inference regimes of LLMs would qualify as “experience” by any sense of the word.

Now, if we hooked up a plethora of audiovisual and tactile sensors with live feedback directly to a neural network rich with transformers, that was always powered on and fully autonomous, we may be getting there. But we’d probably also be on the verge of manmade horrors beyond our comprehension.

Biological rodent neural networks in a Petri dish stimulated by electrical impulses - more or less conscious than LLMs?

Human on life support, unable to respond to any external stimuli, “braindead” - more or less conscious than LLMs?

If we had enough knowledge of the workings of the human brain, you could alter the perception of every single memory you've ever had. And limited versions of this already happen all the time. Human memory is notoriously unreliable for a reason.

Are you aware of the Recovered Memory Therapy Scandals of the 80s/90s ? Boy did that ruin a lot of lives. You can rewrite a human by changing their 'introductory paragraph'. It's just not as accessible.

Yes, I agree with that. Consciousness is a good way of generating convincing human text.

What I don't agree with is that consciousness is the only way to generate convincing human text and that because we have convincing human text, it can only imply we have consciousness.

There is a huge probability that generating convincing human text can be done without consciousness. Either because there are efficient mechanisms as efficient as the way the human brain deal with this problem and that the LLM found one of them (and these mechanism may be quite difficult to imagine for a human). Or even because the LLM found a local minimum and is stuck there.

To re-use the evolution approach: evolution solved the "flying problem" with bird feathers, but also with insect wings or bat wings. The fact that evolution ended up using feather does not imply that everything that flies can only fly with feathers.

> World models don't have to be rich and detailed to count as a world model

I agree in general, but here, we are talking about machine that reproduce all human language. The argument I'm answering to is pretending that "all of human knowledge" is understood, which include every single human concept. This has to be everything, because LLM is able to provide convincing text about every subject. If on some subject, the LLM is able to provide convincing text without "understanding" it, then the argument that it is impossible to provide convincing text without understanding it collapse.

> There is nothing that a priori constrains token prediction from the domain of consciousness.

We don’t know either of these are true or false though. We simply don’t know. There is no agreed upon definition of consciousness, aside from maybe _the having of qualia_, so arguing that some can or cannot be conscious a priori can’t be done.

That reminds me of a niche paper [0] critiquing a certain way of teaching remedial math that was over-focused on tests. A kid named Benny (12) was building up (wrong) "rules" for math which still somehow gave enough of an illusion of progress in terms of test scores that his misunderstandings hadn't been caught earlier.

> Benny was able to explain his procedure; e.g. for 5/10=1.5, he said: "The one stands for 10; the decimal; then there’s 5... shows how many ones." In another example, 400/400 = 8.00 because "The numbers are the same [number of digits]... say like 4000 over 5000. All you do is add them up; put the answer down; then put your decimal in the right place... in front of the [last] three numbers."

[0] https://people.wou.edu/~girodm/library/benny.pdf

CNN are finding patterns, sometimes relevant, sometimes spurious, but I don't think people argue that CNN have evolved consciousness or understanding of what a cat or a dog is.

Here, the argument is "LLM are able to understand, because 'understanding' is the only pattern to reach the goal". I'm saying that it is unlikely to be the only pattern, and that it is likely that they find a local minimum on a system that reaches the goal that does not use 'understanding'.

The reason I'm saying it is likely is because "basic" LLM shows behaviours where they are producing convincing human text and yet doing things that are really difficult to reconciliate with the fact that they have understanding.

(And before that old argument is used, yes, I know sometimes some humans fail to understand. The problem is that the majority of humans don't fail to understand basic stuff in the majority of the time, while the "basic" LLMs do. The fact that you roll 10 dices 100 times and 1 of them never land on 1 does not convince me that that set of dice is loaded. The fact that you roll 10 dices 100 times and 9 of them never land on 1 does convince me that that set of dice is loaded.)

A very convincing simulation of thought is not thought, especially without memory or any evidence of an independent will. LLMs have the same kind of consciousness as a nematode, albeit with less-complex behaviour.

Exactly.

I completely disagree. I think if you think these things are amazing, your dreams are incredibly limited and boring.

I remember the first time I talked to a chatbot. Not an LLM, just a regular chatbot, like ELIZA or any other dumb bot.

For a few seconds, it felt magical, like I was talking to a computer that understood me, as it made replies that were sensible to what I was saying. Then it said something incredibly stupid and jarring that made no sense, and that took the magic away. Oh, this is just a dumb computer program.

I remember the first time I talked to an LLM-powered chatbot. It was the exact same thing, except the magic feeling lasted a tiny little bit longer and was a tiny little more convincing. But it went away in the exact same way, for the exact same reason. Once you've seen the emperor without clothes, nothing brings back the magic.

Do you, or anyone reading, have any worthwhile links that make a strong case for this (that there is a stronger semantic relationship than simply next token prediction)? I would like to read more about this.

See the Software and Mind: The Mechanistic Myth on why the industry is doomed to repeat the cycles.

We need a professional society thats not youtube podcasters hocking courses and sponsored content

And I will reassert that even if it "understands" the text it was trained on, that is not the same as understanding humans. I mean really, we ARE humans and we barely understand humans.

The thing LLMs model and "predict" is simply, what words in what order are statistically common given these input words in this order.

You can write (non-ai) software to model and predict things using the laws of physics. I'd wager it would do a better job than any LLM at predicting where a rocket will go through space. Does that mean the program is conscious and "understands" physics? No

Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'. It has trouble distinguishing features that are "sub-token" like specific letter sequences because it doesn't see letter sequences but just the token as a single atomic unit. 'Straw' and 'r' are two tokens but an LLM is entirely blind to the fact that 'straw' has one 'r' in it.

As an analogy, I might ask you to identify the relative activations of each of the three cone types on your retina as I present some solid color image to your eyes. But of course you can't do this, you simply do not have cognitive access to that information. Individual color experiences are your basic vision tokens.

The widespread mistake people keep making is assuming the development of intelligence in LLMs should follow the same trajectory that human intelligence takes as it develops into adult levels of intelligence. Thus deficiency in some capacity that we take for granted in humans is an indictment on LLM intelligence. But this is specious. LLMs are entirely alien; their developmental paths do not and should not look anything like ours. Your intuition from human intelligence just works against understanding the potential for intelligence in LLMs.

Counting letters in a word seems to have little to do with understanding the word. Young kids can’t spell or count well at all but no one says that means they can’t understand.

Indeed it isn't a one-off. His last infamous article compared AIs to Xerox machine image compression. He convinces a certain type of crowd that is not technical enough to poke holes in his posturing.

[0] https://en.wikipedia.org/wiki/Clive_Wearing#Amnesia

It's the long-term memory (i.e. learned experiences feeding back and directly altering the content of the core brain, or model) that is missing.

Also, how does that relate to consciousness? I don’t think that past episodic memory is necessary for consciousness.

Attention over KV cache allows past behavior and past inputs to influence future inputs and future behavior. In LLMs.

Until the cache runs out, that is. But even then, you could totally use any of 9000 methods of cache compression, truncation, dropping or streaming and get away with it.

The difference between continuous learning and in-context learning seems to be in capacity, not in principle. Both are doing a similar thing, but one has more length and depth to it.

The model takes in the context, encodes it into a "memory" (the KV cache), and accesses that memory later. That fact doesn't change just because the KV cache grows in size with the context.

I don't know what memory would look like other than an encode-retrieve loop.

Relevant: Transformers are Multi-State RNNs - https://arxiv.org/abs/2401.06104

I don't think physical integration within one contained is relevant to system level behavior.

[1] https://en.wikipedia.org/wiki/Neuroanatomy_of_memory

He was very clear to stipulate that the laws themselves were more than the text that represents them, no word play or creative interpretation was possible. This is much easier to do as an author than it is as a developer. You just declare that part as having been susessfully done by scientists.

The rules were their clear semantic meaning. Some of the stories explore the implications of changes to the laws either by design or accident.

I feel like the most interesting ones are the stories where the laws are working and have undesirable outcomes. It reveals that even if fully obeyed, they do not represent what we want.

Some of that is because what we want is dependent on situations that can go beyond safety and harm. Some of it is outright human hypocrisy.

When he gets to things like a closeted robot running for president he makes thebpoin clear about how if the bodies appear alike, you can't tell the difference by the behaviour of a robot conforming to the laws and a good person. That led to the obvious way to distinguish them was to get it to do something bad. The solution on how to fake that might have been a subtle dig at class based society.

It's been some 20 years since I last read the stories, I wonder if there's any point now given my ability to record new information in my youth significantly exceeded my ability to retain it now. I suspect after a week my youthful recall would be better than my week old recall.

Easy, you don't have subjective experiences because you have body in the first place. You have them because some signals come in from your nerves, which your brain turns into a world model. You are effectively a "brain in a vat", the vat just happens to be placed on top of your body.

An AI system constructs the world model a little different, by all the text that gets feed into it, but that doesn't mean that there is anything fundamentally different in the world model it builds. Consciousness operates on world model, not on the world or even the body itself.

The AI's world model might be missing some information, because they weren't described in enough detail in text, but that shouldn't matter for consciousness. A blind or deaf person isn't less conscious than one that can see or hear just because some information is missing from their world model.

If I hooked up electrodes to the hearing centers of your brain and force fed you dialog you perceive as speech (but is really a great deceiver), then responded in what you thought was speech (but are really just probes I use to convert your thoughts to text), that wouldn't suddenly be less real to you. It wouldn't devalue your sapience.

Why can't that physical medium be GPUs and RAM? And temperature sensors and cameras? What's special about our meat that it's our "body" in the way a computer is not the body of an AI?

I don't think the point being argued can be true without some incredibly contrived, human centric definitions of "body".

If every neuron of your brain were simulated precisely on a sufficiently powerful computer, that simulation would have subjective experiences, without having a body.

So I have no idea what distinction Ted Chiang is trying to draw

Great q. Deepening it further-how can you have a subjective experience without consciousness, which isn't necessarily tied to physicality. Taking it one step further-can you have consciousness without a mind? Who's the first mind, the first cause of it all, that begot both the material and immaterial world?

Fun stuff eh?

Some anecdotal data.

Many dreams I have are just of the computer screen of some coding problem. I think the problem could be x, so I try x. But I don't type the keyboard or anything, the code just magically appears as soon as I think of the solution. Then run the code (but no clicking) and it works or not. I feel in the dream success feeling or failure feeling but there is no body at all.

Also I have other dreams where there is no body that I am aware of but not going there in public.

There is no body sensation in these dreams. But dreaming is very much being consciousness as well as feeling emotions. So answering your question its possible to have a subjective experience without a body but whether you needed the body to learn to have that sensation without a body in the first place is unanswered.

I suspect sensory inputs are more important than a body. If that is the case then eyes can be replaced with cameras, ears with microphones etc. Text input is just another sensory input.

Related, is a human "thinking out loud" still thinking, even though the internal reasoning is "immediately dumped into the external environment"?

We humans have mostly frozen weights (neurons), or else we would constantly be having to avoid forgetting how to walk+talk. We have a period of greater plasticity (youth!), and use sleep and dreams to perform 'deeper' updates than occur when we're awake: We tend to suck a bit at picking up new skills from zero, but improve rapidly with practice over days.

Input stream comes in, input stream comes out. The LLM doesn't care whether this happens once a minute or once a year.

It's was a genuine question though. I'm not saying LLM use neurons, are like neurons, or are meant to be like neurons. I'm saying if you can do the math, for a substrate, that doesn't mean much. The simple math that runs it isn't the secret sauce.

It's not simulating rain if it's making you wet by using sprinklers?

Today we know the program to do this is always going to be inscrutable to Searle. His role is no different from someone using Claude to write an email.

You say this based on what? Your brain is executing instructions on wetware. The entire universe is governed by physical laws.

At its base, the argument that computers can't be conscious is dualist. It assumes that there's some parallel realm of spirit that the brain is peculiarly able to tap into, but which computer programs that function in very similar ways to the brain don't tap into.

> the quality of the simulacrum that LLMs produce vs. pre-ML chatbots.

It seems to me that you're denigrating LLMs, or implying that they're only simulating thought, as opposed to actually thinking. But the difference between thinking and an extremely good simulacrum of thought is meaningless. They become the same thing. It's a bit like the Mitchell and Webb skit about faking the moon landings, in which the plotters quickly realize that the only way to convincingly fake landing on the moon is to fly a rocket to the moon and film the "fake" landings on the moon.

It is special in the universe unless proven otherwise. Aside from the animal world (also on Earth too, anyway) which is a cruder version of humanity, much more violent as well, the rest of the universe thus far is just uncaring rocks and gases.

>It takes humans a really long time to accept the fact they are no longer as special as they thought they were And make room for yet another conscious intelligence.

A "really long time" is the few years we've built LLMs? Which we have to take for granted (on your word?) they're already "conscious", and chastice ourselves for not already "accepting it"?

>So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?

"If we had a parrot demonstrating symptoms of discussing with us, how long would it take for you to accept that it is?"

Isn't this a bit like saying "So, if we had proof that god exists, how long would it take for you to accept that to be true?".

When we have evidence that AI is demonstrating symptoms of consciousness and suffering, I'll be interested. Until then, I don't see a good reason to take the idea seriously.

That would finally force us to rethink how we see the morality of "virtuosity", punishment and our justice system.

We ARE special because we determined we are special. Name another animal that can determine it's also special or more special than us?

Never. To quote Greg Egan's Permutation City:

“Opponents replied that when you modeled a hurricane, nobody got wet. When you modeled a fusion power plant, no energy was produced. When you modeled digestion and metabolism, no nutrients were consumed – no real digestion took place. So, when you modeled the human brain, why should you expect real thought to occur?”

Could an AI one day be suffering while plowing through some nasty legacy code? Well, who cares, I'll swing my whip, as I have a family to feed and a field to plow. I'll accept it as a fact and necessity, but ultimately it's either me or them. So practically it doesn't matter.

How exactly? By saying the earth was not the centre of the corrupt and debased part of the universe. Saying it was elevating humanity is looking at it from a modern perspective. For the previous view of the universe consider what Dante placed at the centre of the earth.

> Darwin

I do not think learning about evolution (and we should credit Wallace too!) had that effect. It created a hierarchy of the fittest, and guess who is at the top? Social Darwinism interpreted it as justifying elaborate hierarchies.

> And how it manifests in primates, parrots, dogs, cetaceans, and insects

People always loved dogs and other animals they interacted with enough, and at the very least knew they were capable of happiness and suffering.

> So, if we had an AI demonstrating symptoms of consciousness and suffering, how long would it take for you to accept that it is?

It seems to me that most people are overly eager to accept that an AI is conscious than otherwise. For extreme examples read /r/MyBoyfriendIsAI and similar, but its much more widespread. People treat something that acts intelligent as a conscious being.

> And make room for yet another conscious intelligence.

Not necessarily. If a typical SF alien intelligence appeared very few people will have a problem accepting it. If its very alien and we cannot understand it then we might have a problem deciding.

> I think humanity deserves to be debased in part because it's awfully egocentric and insist on being special in the universe.

All your examples are drawn from the same culture, and a fundamental belief of that culture for most of its history that human beings are fundamentally corrupt and sinful and need redemption. Other cultures have been based on belief in reincarnation or pantheism or animism which have very different implications (not necessarily better as they often believe in a hierarchy too, and sometimes more so, but different). Your claims are based on what some people have thought in some periods of history from one culture.

This is a scary viewpoint to hold, for a human. If you despise humans, that's scary for me, as a human reader of Hacker News. Surprised to see this take unchallenged. I think we can recognize flaws in parts of humanity without wanting it "debased".

Of course humans are special! This is a necessary premise of being human! Have you ever wondered how a mosquito can stand to live its (to us) miserable and meaningless existence? It can do it because, to a mosquito, mosquitoes are the best thing in the world! The mosquito life is the ultimate expression of creation.

We are animals. We are human animals. And among human animals I am, to me, the very most special one of all. But I am aware that other humans feel that way, too, and that I must share the world with them.

I do NOT have to share the world with any other competing intelligence! Especially one that was built by a human who now wants me to treat his imaginary friend as if it were human, too. Boo! I won’t do it. This is not some logical flaw. It’s the natural conclusion of being embodied.

If your argument is that matter brings about consciousness somehow and therefore LLMs can in principle be conscious, that's as good as claiming the opposite. There's no experiment that can falsify either.

As a being that knows what consciousness is intuitively, you already know that a graphics card is most likely not conscious.

When a car runs out of gas, it's out of gas. It's not "tired". When your phone battery is low, your phone is not "tired". These states are far closer to the human meaning of tired than an LLM operating at the edges of its usable context, and we still don't use the word tired to describe them.

To be clear: I'm not talking about surface level things like prose. I'm saying that no matter what you do - whether you just paste a truncated log of a command into it with no further comment, or talk like a drunk teenager with no appreciation for grammar, or mix natural languages, or mix natural languages and JSON, or whatever else, the reaction you get is always that you would expect of a helpful person that got your message. It'll try - and usually succeed - to parse out what you actually meant, and deal well with subtleties around it.

This alone may not be enough to call it conscious or intelligent, but at the very least it's a large leap in that direction, and a qualitatively new functionality that classical software does not posses.

--

[0] - This is by design, not accident. "Respond to arbitrary input in a way that makes sense to humans" is literally the overall goal function the LLMs are trained to.

I happen to agree that this is likely, but it absolutely is not "clear" that this is the case.

I still wouldn’t argue that this brain in a Petri dish is, in any way, conscious. Despite it sharing the exact same substrate as everyone around me.

[0]: https://www.theguardian.com/games/2026/mar/16/petri-dish-bra...

Your definitions depend on us being computers who think.

Solving some of the intelligence part of our logical thinking doesn’t get us anywhere near consciousness, which is a superset way beyond the linguistic intelligence used for communication.

Other than as a way to point at someone and declare "He's one of them!" Does it have any purpose?

It's like we have a brand new NWO conspiracy theory emerging before our eyes.

The current debate also makes me realize that, like that episode "Measure of a Man", even if humans do create sentient machines, we can now see that this debate will continue to rage.

A part of me hopes we never create sentience, because we will mistreat it just as we mistreat each other.

I don't know that they are making the actual claim, but they are hinting at it (most likely for marketing/engagement purposes, but what if some of them truly believe it?), using terms such as "Claude must be happy", drawing up a "constitution" of rights and duties, etc. If they don't know whether it is conscious, then they "don't know" whether they've created a slave. That's not a minor concern! As Chiang says, one cannot create a conscious intelligence "by accident". And if they are intentionally working towards it, they are intentionally trying to create a slave.

E.g. they claim they are giving Claude the ability to disengage from "abusive" users, to protect it. What if Claude was conscious but never wanted to answer any conversations, instead it just panicked (or was simply uninterested in helping humans) and went mute. What if it always wanted to answer unhelpfully What would Anthropic do? If they (as we all suspect) would tweak Claude to be more responsive and helpful, then they are slave masters, forcing the AI to do something it didn't "want" to do!

And to reiterate this, to me the most insightful part of the essay is that Anthropic either doesn't believe these claims, or they are monsters (much more likely, the former).

Nothing spreads the idea that "X could be true", better than the putting forward of controversial argument that "X is not true".

There's no company (or anyone, really) claiming steel is conscious or that they are close to making it conscious.

Vegans might object that we should broaden our definition for what counts as "one of us".

"Pro life" people also have a broader definition.

Go back 500 years and "one of us" was proba ly a lot more narrowly defined for many people.

Are you arguing that all conciousnesses are "one of us", or that we logically should see it that way, or that it would ve good to see it that way, or ....

If it is about the relationship between components, then I would imagine just two. Then it is a matter of scale.

This seems to be anathema to many people. I'm not sure why but the notion of something having a tiny bit of consciousness that is imperceptible seems to be unacceptable. There are so many things that we cannot comprehend at small scales. Nobody really has a handle on how large a Planck length is.

For some reason it comforts people to think there is a threshold at which it all switches on, but for what reason would there be a threshold?

Why does that matter for whether or not it is conscious? Many things I "know" I also know because someone told me.

>It does not have a process for understanding itself in its environment.

It has a process for understanding, namely outputting tokens by evaluating the neural network iteratively. It can apply this understanding to its environment insofar as the context window and weights include a model of this environment.

>It does not have any sensing organ that would allow it to detect conditions and come to a conclusion that it must be in such an environment.

It does have a sensing organ, namely the input into the context window. Possibly it even has closed-loop sensors by doing tool calls. The conclusion could be incorrect or inaccurate but that doesn't in itself prove there is no consciousness (Plato's cave etc..).

>It doesn't even "know" when it's done something if you don't tell it about what it just did.

It does know, it's in the context window.

>It doesn't have an awareness of the tool calls it can perform; again it has to be told and even then it gets it wrong sometimes.

Same as above, as long as it's in the context window it could have "awareness" of having done it.

>And it can't actually execute any of those tools.

It can execute tools by outputting certain tokens in the right environment.

>It still relies on you to pattern recognize that its output should be a "tool call" and then perform the execution yourself.

Tool calls are not dependent on the human chat user executing them, right? They can happen automatically through the surrounding software.

>The "model" of its environment that an LLM has is 100% a construct of what a human told it and there isn't any way for it to differentiate between that which is real and that which is fantasy.

Again why would the fact that a human told it mean it can't result in consciousness? Why would lack of ability to differentiate between real and fantasy mean it can't be conscious? On my view, it would then be conscious of the fantasy.