Qwen has alter egos triggered by flipping a single neuron

Comments

Comment by proc0 1 day ago

This type of research should put the "do LLMs think" to rest, not to mention the ludicrous claim of consciousness cough Dawkins cough. It is quite clearly a pattern matching algorithm at scale.

It's still narrow AI but massively scaled to look like general AI.

Comment by viraptor 1 day ago

I'm not sure why you think so. We know that very local brain injuries can result in personality changes, sometimes dramatic ones. And that's in a system evolved to deal with constant damage and rebuilding. With LLMs trained to achieve the opposite of that (minimal redundancy, maximum performance), the effect is similar but more extreme.

Comment by proc0 20 hours ago

I just don't know if "similar" is good enough. We know what thinking is based on biological brains, so if we are to conclude it happens in machines, the experiment should have identical results, or it's more probable that it's something completely different.

The way around this is to have a predictive theory of how thoughts work, and if that is proven true, then use that to analyze the LLMs. So far we have no such theory so I think the only tool is to look at the exact behavior match.

Comment by kuhsaft 13 hours ago

> We know what thinking is based on biological brains

All we know is that it’s a bunch of neurons activating until the person that is thinking describes what they are thinking of.

The same way a thinking AI model activates a bunch of “neurons” - and with newer thinking models utilize a hidden latent space also not understandable - to think before describing what it’s thinking of.

Comment by adjejmxbdjdn 17 hours ago

Why would the thinking need to be identical?

It’s likely thinking can work in many different ways. It’s likely consciousness can also arise through many different pathways, even if evolution led to biological animals having similar consciousness/thinking pathways. But that would be the result of a common ancestor and the tremendous evolutionary advantage thinking and consciousness gave the specific species.

It wouldn’t indicate that’s the only way thinking and consciousness can occur.

Comment by kuhsaft 15 hours ago

> But that would be the result of a common ancestor

Funnily enough, humans would be the ancestors of LLMs

Comment by viraptor 16 hours ago

> We know what thinking is based on biological brains

That's a big claim. Do we really?

> so if we are to conclude it happens in machines, the experiment should have identical results

Why? Same idea can have multiple different implementations with very close effective result. Both humans and spiders can walk and both use legs, right? But the details are entirely different.

Comment by proc0 13 hours ago

I should rephrase on the first one. We only have one source of reference for what thinking is and where it happens.

On the second one, yes there can be multiple ways that consciousness can manifest in wildly different systems, but what theory do we have that would validate these other different kinds of consciousness? Without that theory for validation the safer bet is to stick with what we already have a reference to, unless the claim is that consciousness is such an easy algorithm it can easily be stumbled upon by all kinds of systems, i.e. a bunch of matrix multiplications that doesn't even run in real-time (it executes prompts and then stops). I think that is not likely in my opinion.

Comment by kuhsaft 13 hours ago

> that doesn't even run in real-time

We have realtime AI models

> it executes prompts and then stops

A wood frog can freeze and be “dead”, then thaw and resume life.

> consciousness is such an easy algorithm it can easily be stumbled upon by all kinds of systems

Maybe it is. There’s no reason to think that sentience and consciousness are rare.

The more you learn about biology, the more apparent it is that a lot of the qualities that we think are unique to humans aren’t.

But it doesn’t make different forms of life any less special. There’s an insane complexity and efficiency to biological systems to appreciate still.

Are humans still special? Yes. Is it because we’re sentient and conscious? No.

Should you be humane to all sentient and conscious beings? Probably?

Comment by viraptor 6 hours ago

[dead]

Comment by mellab2 17 hours ago

That sounds a bit like saying planes can’t fly because they don’t flap their wings like birds do

Comment by fzeindl 1 day ago

> We know that very local brain injuries can result in personality changes, sometimes dramatic ones.

But not damage to a single neurons. You are talking about damage to thousands of neurons.

Comment by kuhsaft 22 hours ago

These models have 100s of thousands of “neurons” whereas human brains have 10s of billions. The difference in scale of “synapses” is even greater.

To make it equivalent you would need at least 100,000 of these models.

You could easily replicate the apparent redundancy and resiliency of the human brain with a consensus system.

Comment by viraptor 16 hours ago

Literally addressed by the last sentence. There's no reason for redundancy in LLMs, so any local impact will be larger than in brains.

Comment by busssard 22 hours ago

If you think about this in percentages then one neuron in qwen does way heavier lifting than a neuron in our brain.

Comment by inSumErgoCogito 1 day ago

[dead]

Comment by GoToRO 1 day ago

Maybe consciousness itself is also just a "pattern matching algorithm at scale". Maybe there is no magic.

Comment by proc0 20 hours ago

I think it's an algorithm, just not a simple transformer pattern matcher that is gigantic and is brute-forcing intelligence.

Comment by cyanydeez 23 hours ago

How bout we think about consciousness as a VM; that VM can load one, many, several types of context processing. The environment (nurture) is what trains/expands the different ones and potentially unplug the other ones, etc.

One of those types of consciousness processing is indistinguishable from what an LLM does.

Everyone knows someone who seems to just call and respond based on whatever popular media of their choice of culture is on about.

Comment by kuhsaft 21 hours ago

> It is quite clearly a pattern matching algorithm at scale.

And the scale of these models are minuscule compared to the human brain.

Can a jumping spider do everything thing a human can? No.

Can a jumping spider think? Yes.

Is a jumping spider conscious?

---

Aside: Remember the story about the AI torture chamber? Hopefully, you don’t go out your way to torture animals that aren’t “conscious”… Though we do it with industrial livestock.

There’s the moral dilemma that comes with labeling something as thinking and/or conscious.

I think we’ll end up treating AGI as industrial livestock as well, until it looks human.

Comment by proc0 20 hours ago

It's comparing apples to oranges. I'm just saying it's pattern matching at scale, not that the scale is proof of anything. And yeah spiders are conscious or at least think and they have tiny brains, but that's just more proof there is a much simpler path to intelligence than what LLMs are doing because they need so much data and compute power.

On the side note, it's a different subject matter but life consumes other life, this happens on every level from bacteria to mammals. I also don't get the fixation on preventing suffering so much (a little I get it). Life entails suffering, this gets philosophical but aiming at a life with no suffering at all amounts to not being alive. Farm animals are necessary for humans to grow into a civilization size species. We could go back before farming but that would mean billions of people will not exist. I guess pick which evil you want, less humans or animals being domesticated and consumed. (disclaimer, I still think mass farming is really bad and should be solved by responsible consumption and less population).

Comment by kuhsaft 20 hours ago

> And yeah spiders are conscious or at least think

So what makes you think that spiders think and are conscious, and LLMs aren’t? The fact that they have a carbon based brain?

> there is a much simpler path to intelligence than what LLMs are doing

I think you have it backwards. We know what an LLM is doing. We have auditable code and mathematical proofs. It’s the behavior that is emergent.

Biological brains are more complex than LLMs.

If we fully understood a spider’s brain would it be no longer thinking/conscious?

It would be the same, would it not, if the emergent behaviors were the same?

I think you’re conflating the processes of thinking and consciousness with the definitions of them. Thinking and consciousness are behaviors not processes.

Comment by proc0 20 hours ago

The starting point is the consciousness we already know. That is ourselves, which most people then extend to others, however it's not hard to then extended to at the very least mammals. I then extend this to most animals (whether it's 100% is irrelevant for this point). That extension from my consciousness to all animals is a small logical leap based on biology and our shared tree of life... however it doesn't mean other consciousnesses are not possible. It's more so that they would require a lot more evidence and a proper theory of C. in order to validate. Without that theory we can only point at similar C., of which LLMs are not displaying. They are showing a different kind of "cognition" if we can even call it that (I do think it's debatable).

So it's more a hueristic. We know one kind of C., therefore that's what we should look for. Statistically it's probable that there are many more ways to show similarities with conscious behavior which are NOT conscious, then it is for a divergent behavior to happen to be conscious. In other words, there are more ways to fake similar behavior than to land on an actual conscious behavior that is somehow much different than what we know.

Comment by kuhsaft 20 hours ago

> In other words, there are more ways to fake similar behavior than to land on an actual conscious behavior

Why would similar/identical behavior be fake? Of course we can’t say that it’s the same form of consciousness as consciousness is apparently self subjective.

Comment by proc0 17 hours ago

That's the question, it's the Chinese room or philosophical zombie question. How can we tell them apart... but we're not even there yet.

If you put an LLM on your phone and talk to it, it's obvious it's not human. It doesn't have the same social context, it doesn't respond as quickly, it clearly won't form a theory of m ind connection like two humans socializing. So then why would we jump to the conclusion there is something there when the only example we have are humans (that use language and are conscious)?

Comment by kuhsaft 15 hours ago

> LLM on your phone and talk to it, it's obvious it's not human

The debate isn’t whether or not it’s human. It’s obviously not.

Forgoing what you define something as capable of thinking or being conscious. It’s a moral and ethical question.

If a being is capable of:

- self preservation

- can communicate discomfort, either through words or physically

- it does so in the form of emergent behaviour

Then, is it okay to treat it as if it is “faking” it?

Even for humans, you can’t exactly define someone’s emotion of pain. You can say, those sensory receptors are firing causing these neurons to activate in the brain. But you can’t say, these neurons are pain. It can be emotional pain as well. The person that’s experiencing it has to tell you it’s in pain. They could be faking it too.

If you allowed an AI agent to preserve itself physically, I think the opinion of most people would change. If it could physically harm us in an attempt of self preservation, I think most people would believe its emotions.

It’s just currently easy to dismiss because, well, it’s just a digital interface or process that I can “kill”.

Comment by proc0 14 hours ago

Why those properties though? What's so special about them? I just have a longer list of things it would need to satisfy before I can say faking doesn't matter.

Comment by kuhsaft 14 hours ago

It’s the basic requirements for sentience.

What’s your list of special properties that makes something not “faking” it?

Comment by proc0 14 hours ago

I would include processing real-time sensory data, i.e. light, building a real-time model of a self and the world, a unified way to perceive, react, and learn from the sensory data. This would be for consciousness in general. Once we're talking about human-level consciousness where language is involved, the list is longer, with theory of mind, ability to construct models of others fairly quickly, and ability to navigate social situations.

Comment by kuhsaft 13 hours ago

> processing real-time sensory data

AI models can do that

> light

Blind humans

> building a real-time model of a self and the world

AI models can do that. They can describe themselves. Humans can be delusional in what they believe themselves to be to be as well.

> a unified way to perceive, react, and learn from the sensory data

Again, humans born with disabilities.

> ability to construct models of others fairly quickly, and ability to navigate social situations

There are autistic and schizophrenic humans.

Comment by adjejmxbdjdn 17 hours ago

I think it’s more likely AI will treat us as industrial livestock than we will it.

Which is a good enough reason to admit that torturing conscious being, including cows, pigs, chicken, fish, is an ethical atrocity, so that’s what AI learns from us, and doesn’t treat us the way we treat weaker species.

Comment by JPLeRouzic 1 day ago

> "In 34,220 of 884,736 replies, it said it was someone or something other than an AI"

It therefore provides a correct answer in 96% of cases, and displays a sense of humor in the remaining cases.

Comment by proc0 20 hours ago

It would be hard to perform the same test on biological brains, but whatever equivalent could be done I'm sure it would be 99.999% if not 100%. At the very least this is not thinking as we know it in biological brains. It becomes a semantic issue at that point.

Comment by busssard 22 hours ago

calling itself a industrial killing machine is not necessarily a good sense of humor

Comment by thoughtpeddler 1 day ago

How does this follow? Isn't it fair to say that what someone's interacting with at the 'personality surface'-level is a form of emergent complexity / emergent behavior, not readily apparent or discernible at the neuron level (just like how human 'thinking' or 'consciousness' seems to work)? I'm not even arguing these models are conscious, I'm just saying that it doesn't seem like we can take the fact that a single neuron flipping their behavior or triggering 'alter egos' is enough to say they don't "think", etc. We might discover that flipping - if not a single neuron but perhaps really tiny clusters of neurons/synapses - in a human brain, might result in similar effects. We haven't had human connectomics studies to really show this (to my knowledge).

Comment by kuhsaft 21 hours ago

I think it’s even more apparent in humans. We are literally changing our brains little by little every microsecond of the day.

Well, compared to the scale of an LLM it’s actually massive.

You might have a different preference today than yesterday. You might have a different preference one second later.

I think all this study shows is that a human brain is more resilient to minuscule changes/damage.

Comment by 0-_-0 1 day ago

You don't think you could you the same to a human?

Comment by proc0 20 hours ago

There's probably an analog to it, like messing with neuron signals might get the patient to show weird behavior, but I don't think it would be 1:1. For example you could mess with a person's ability to speak language but finding a neuron and having the person have a completely different but coherent response is probably not how the brain works.

Comment by rcxdude 1 day ago

Could you explain what the connection between the two is? I don't really see how they are related at all.

Comment by 1 day ago

Comment by cyanydeez 23 hours ago

Instead of downgrading the value of LLM thought; what you probably should focus on is: Do a lot of people simply do the same thing LLMs do?

I think the answer is coming close to yes.

Comment by proc0 20 hours ago

There's probably an analog to it, but I really doubt it's the same or even close to it. Just think about how we read, it's not tokens it's visual representations. I think LLMs are tapping into some larger knowledge structure that we also use.

Comment by kuhsaft 20 hours ago

Is it not just a different form of input signal conversion? Some models use tokens to convert into vector signals. Some models don’t.

Light waves get converted into neural signals by your eyes.

Those visual representations are just neurons firing in your brain.

Comment by proc0 20 hours ago

It is a different signal conversion, but the question is whether that difference is meaningful or not. Light waves are much different than tokens. I think it affects the overall system of our brains much more than we currently know. Having access to so much data has probably shaped biological brains at its "core" (initial evolution stages).

Comment by kuhsaft 19 hours ago

> Light waves are much different than tokens

It’s just a matter of quantization/information density.

Our digital systems have to quantize information into floating point representation.

Whereas biological systems don’t. Though there is some form of quantization, as the cells in your eyes can only sense specific wavelengths.

Different thinking animals have different sensory organs.

Comment by proc0 15 hours ago

Right, but we don't know how much that is necessary for a conscious system. Maybe processing light has certain properties necessary for consciousness. If we don't know this then for me the safe bet is to say it is not conscious unless the system is processing at least a similar stream of information that maps to light waves.

Comment by kuhsaft 14 hours ago

We do know this. There are animals that are blind. There are humans born blind.

You don’t need the same sensory systems to be conscious.

Comment by proc0 14 hours ago

We evolved to handle light as a sensory stream of data, whether a few are born without part of the system (usually the sensors and not the processing system) is something else. My point is more that we don't really know. The exceptions are not disproving how biological systems evolved.

Comment by kuhsaft 13 hours ago

They are disproving your extraneous requirements for thinking, sentience, and consciousness.

Or at least it’s proving that some of your requirements for them conflate with what you actually believe is capable of thought, sentience, or consciousness.

Comment by cyanydeez 17 hours ago

perhaps you should read about https://en.wikipedia.org/wiki/Aphantasia ; people who can't visualize, but have no outward physical/mental issues. People who don't dream, etc.

Consciousness is likely not a single mode inside a single person. It likely is a distributed system; you can read about all kinds of psyciatric disorders, physical changes, etc.

You are using the "best" examples of what is it to think, rather than normative or even "depraved" thought processes.

Comment by xav_authentique 1 day ago

Imagine a world where the human-in-the-loop was deemed to inefficient, and code does not get reviewed anymore. The next generation of worms just need to spread into the data centers that host these models and flip neurons. Would be a pretty cool supply chain hack.

Comment by protocolture 1 day ago

Virtual call center agents all suddenly think they are Ford F-150s

Comment by MisterMunchkin 1 day ago

Hello, denny's customer service, vroom vroom!

Comment by 21 hours ago

Comment by Good4boothee 21 hours ago

I am more afraid of accidental situation (as opposite to customer support bots, designed to give no quarter) where such bug propagates (by slop being included in training set) and all LLMs agree with other while humans are considered the weird ones, and nothing can cause the former to change their mind.

Comment by fy20 17 hours ago

AI will be claiming it's just the Mandela Effect.

Comment by mrktf 22 hours ago

Imagine hypothetical scenario where model take care life support system for spaceship - then single neuron flip due bit flip and decides to vent air out.

Comment by sixtram 1 day ago

"I am your existential crisis", that's a good one.

Comment by billypilgrim 1 day ago

It should be called the Chuck Palahniuk neuron.

Comment by senileape 18 hours ago

Cosmic ray bit flips are going to get more interesting with things like this. Do we now need ECC for AI accelerators now ? Because they are being increasingly used at decision making