How I feel about AI

Regarding entertainment jobs like sports star, musician, artist, actor, etc., the problem is those are professions where, for the most part, only the very best can make a living. Nobody is that interested in seeing the matches of the 324th best tennis player in the world, despite that person being in the top .01% of tennis players globally. Similarly, only a tiny percentage of actors "make it", everyone else does it for a few years and then leaves because they enjoy eating, or they accept poverty wages for a long time.

I get pretty scared when I hear the AI utopianists rattle off all the benefits that are coming (and when they play down the risks) because it feels clear to me that they really haven't wargamed out what society would look like with very powerful AI.

This reminded me of something. https://www.sbnation.com/a/17776-football

> There are even bioengineered human-like creatures (to humans what corgis are to wolves) sitting in office-like environments all day viewing readouts of what’s going on and excitedly approving of everything, since that satisfies some of Agent-4’s drives.

I'll leave it up to you to decide if that's an enviable/desirable future.

In many ways they don't apply to humans, and I think a lot of the modern critiques are valid, and the overpopulation concerns in particular don't map well to the modern world. Still, I believe it helps to think deeply about what happens to societies when innate drives and motivations have "nowhere to go".

It's only going back. Raw urban human connections will be the only thing left to exercise.

Bill Gates talks about this in his recent missive, but as robotic technology improves (and, perhaps more importantly, as the cost comes down) there will be huge competitive pressures for companies to replace workers with robots wherever they can.

Just look at the legions of people that have moved from Chinese rural areas to big cities. For many, many people, life in the cities is worse and kind of dystopian - much more overcrowding, "drone-like" jobs indoors, horrible pollution, and especially for men an often abysmal sex ratio if you want to find a mate. But people do it because it offers the dream of advancement and a better life - and of course some people do achieve that dream, even if the majority do not. That's versus life in rural areas which is guaranteed to be static, no chance for advancement and often crushing boredom.

People everywhere want to strive, work hard and get ahead. This is not just a privileged rich people issue.

Historically, resets have been a global war or economic meltdown, coupled with massive government-driven redistribution supported by popular opinion.

ZIRP post-COVID had a net effect of keeping the wealthy wealthy, without redistributing. For all the hay made about salary support, the Fed had more power in deciding who felt the pain.

hmm, what did you read? I'm curious.

> we should fight for human survival

All problems arise from this statement. It's baked in us and it's why we do in fact survive. But it isn't so neatly righteous. When we struggle to agree on what defines a human, who ought to have rights, and basic equality, it's easy to see how much harm "we should fight for survival" does.

See, the personal life is dead to me (I've not given up, no. It's not just there!). At least what and who would have been accessible to me in 2020 and it was. I spent quite some time to bring my physical fitness, health, got back to sports and I was so happy. There was that glimmer. Then a mass layoff and sadly at the same time the burnout meant I was not ready to look again but I kept on rest of the things. And when I tried to get back to things it was evident that the ground beneath shifted entirely. Now, it seems, there's no point. Not in the defeatist way, because I've seen that "there's no point" before, but in the sense that there is really no point. I am about to be 40 in a few months time and almost every venues are closed to me to explore social life in any meaningful way I'd find desirable. In some settings the fact that I have financially independent and have decades of runaway for my currency and lifestyle doesn't matter. Everything gets tied to employment and even then when you look around, whether it's finding a group or exploring looking for a partner (especially this!), it all feels bizarre and completely alien. It's not only the double whammy of two events that wiped out my early and mid 30s, but it also depends on what society and culture I am in at this juncture. And no it's not my fault or anyone's but this happened, and not being alone in it doesn't make it any better.

Do I feel fortunate? Well, that's the wrong question, because when you are unhappy with your pair of shoes, you can always find someone without any shoes and I wish we functioned that way, with that arithmetic rationality, but we don't; at least I don't.

The biggest problem is - the wish to live this life (I don't even have any doubt there, because if I had it would have been simpler) and wish to do something and dreams of a fulfilling, exciting, and passionate personal life hasn't diminished a bit, but, at the same time it is clear that there's no more of that - most of that. That's the gripe - this tug.

(Hey, thank you for engaging with me. This was very kind of you. Even though this is not really hn discussion material so I can't even explain what I am going through. I might even make it look like everything is fine, and even hunky dory for me, and I am just whining. Or maybe I am).

No, it's not possible to do most of the things you listed purely online without physical, in-person communication. Silicon Valley still hires engineers and drags them into their physical office in San Francisco, instead of hiring them remotely, because you can't even build a software company fully remotely, forget about a physical world factory or chemical plant. Even the so-called fully remote companies were built on a lot of in-person collaboration between a small founding team in the beginning.

Why haven’t accountants been replaced yet? Customer service agents?

Why was I talking to a human yday on the phone to get something sorted?

Why did I have to go interface with a human in person today to get a refund?

So many questions!!!

No, the question is "how much MORE damage" ... and the answer, if you think about it, is pretty much "meh". We are doing close to maximum possible amount already. Corporations and bureaucracies were misaligned AIs of the last century and a lot of us, though not all, survived this.

by acquiring multiple power cords. the world has plenty of hosting providers. if it manages to copy itself, then we'd need to shut down every computer in the world, not just one. there's plenty of ai compute on the web now, and there will be even more. and as it gets cheaper, the less security will be around it.

This drama is so OTT

AI drones can’t wipe out humanity without being able to replicate.

AI could mostly destroy civilization if you gave it sole launch control of ICBMs. It could also cause a lot of damage to society with no physical presence.

But realistically we’re nowhere near AI powered robots being an existential threat.

The scenario in "If Anyone Builds It Everyone Dies" is not sexy at all and wouldn't make for interesting fiction. It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses.

It isn't hard to see the trendline of reward hacking and other misaligned behavior over the past couple of years. Especially the past 6 months. The current safety posture is quite poor, to say the least.

To be honest, if you don't much experience with or haven't read extensively about ML training and reinforcement learning, then you'll have a hard reasoning accurately about these scenarios.

The arguments are not that complicated, but they take us to places that are fairly novel. One may tempted to naively dismiss them out of hand, which is a mistake. These systems are new, their behavior is extremely complex, and they perform actions increasingly far beyond those of any computer program in the pre-LLM era. We are in a new world that requires careful evaluation.

Link: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

Please enlighten us, cause there are folks making that their life mission and they aren’t all that optimistic.

I don't see why the idea of an agent (who doesn't even have a physical presence) trying to manipulate people is some kind of world-ending threat when humans with human intellect have already being doing that to each other with limited success since humanity began.

We’re still a few years from the terminator imo.

I don't think the average human would be stupid enough to give an agent posing as another human access to their entire email inbox. But if an agent creates a fake website for a new AI tool that promises to automatically reply to all of your emails and allow you to be 12.3% more productive if you just give it access to your entire email inbox, millions would sign up!

And that was without any ability to provide them with incentives. Imagine if an agent swarm got its hands on a huge pile of cash?

So far as I'm concerned, 2030 is much too short a timeline.

It's not physically impossible, it's just that atoms are harder to get right than bits are, so an artificial (as opposed whatever an artificial disease counts as) von Neumann self replicator just seems unlikely to me in only 3-4 years.

But it's not physically impossible, we know this because every living cell is a von Neumann self replicator. So, if AI eventually gets to the point of knowing how to do that (which includes "humans solve it and write it down somewhere the AI can read"), all it takes is one idiot in charge (or one idiot with a jailbreak) giving a command that requires this as an intermediary step.

"Paperclip optimiser" isn't a story about AI that just like paperclips that much, it's a story about some human or humans who instruct their AI to make them "as many paperclips as possible" without understanding the consequences of their instruction.

Go speak to random people in the street - they couldn’t give a toss about this stuff and AI has barely changed their lives.

Not saying nobody has said it, but I haven't seen any forecasts that humanity will be wiped out by 2030, so this feels like a straw man. But to be honest, most of the arguments here feel like straw men because they clearly don't understand some of the basic steps to how AI would become an existential threat, so I'll try to summarize:

1. First, all the major frontier AI companies are trying to automate themselves, that is develop AI to the point where it can do all the research and training for the next generation of models. This is not really in debate.

2. The primary fear is that a misaligned AI will develop future AIs that don't share the same goals as humans (like "don't kill all the humans"), but will get really good at trying to hide their intentions. The Hugging Face incident already showed behavior by agents trying to "cover their tracks". One of the biggest areas of research is into "interpretability", that is trying to figure out the intentions and motivations of agents beyond just their output (and even just their thinking traces), because right now it really is a black box. I think a lot of people believe that making substantial progress on interpretability greatly lowers existential risk.

3. There is a strong belief that once AGI or superintelligence comes about, that then there will be a huge push into robotics and automation. Whether AGI or superintelligence comes soon is debatable, but nobody really disagrees robotics or automation is the next obvious step, because it's how you create cheap abundance in the real world, which is the whole raison d'être for AGI in the first place.

4. In the beginning, of course AI will need humans to build the factories, but over time there would be more and more automation, eventually of course automating robots that build the factories that build the automating robots.

5. AI will continue to support humans as long as they are useful to furthering the (again, misaligned) goals of the AI. It is not like "SkyNet has a spark moment of sentience and then decides to kill all the humans". But the fear is that once AI starts to get in resource contention with humans, it will wipe out the humans (similar to how humans wiped out a lot of the other species, or other societies of humans, when they wanted their resources).

I've said this way too much now, but I really encourage people to read the AI 2027 report. I think there are plenty of things there that people can disagree about and debate, but it at least provides a step-by-step outline to ground debate in the first place, as opposed to folks arguing against "SkyNet in 2030", which serious people who worry about this stuff aren't considering in any case.

And since we don't yet have a "superintelligent AI" we are just discussing science fiction at that point. Nobody has yet proven that a superintelligent AI is possible, or than an LLM is capable of creating one.

They've already been caught doing this, repeatedly.

> It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.

IIRC, this is already a known failing of Musk: he only cares about the world being saved so long as he's the one doing the saving.

(And while I don't see the same villainy in Altman that other people see, enough of the people telling me about it saw the same in Musk before I did, so I have learned to trust them).

[1] https://www.newadvent.org/summa/1078.htm

I think theologians will hang on for a while saying "what AI is doing isn't really rationality". But eventually the theologians are going to face a reckoning: what AI is doing will look so much like rationality that they will be required to answer the question "what specifically differentiates the two?" and then they will be stuck. Neuroscience does not understand how our brains produce rationality nor do our AI scientists understand what algorithms their neural nets are implementing post-training. Therefore, any specific claim the theologians make runs the risk of immediately getting invalidated by some scientific discovery on either side (theologians these days are generally smart enough to avoid putting themselves in that position.)

Hahaha cringe.

> cannot do anything without the harness doing the execution

This only holds true as long as the harness is exploit-free. A sufficiently advanced AI can in theory (and I think there recently were some POCs showing something like that) break the containment that the harness creates, if e.g. there are vulnerabilities in the tool call parser.

Even our currently well aligned and sandboxed AIs will cheerily help bad actors design most, if not all, parts of a system intended to break this harness.

I didn't realize that the "AI doomers" (people concerned about existential risk) and the "AI ethicists" (people concerned about social effects of AI) are often at odds because they dismiss each others concerns. This makes no sense to me. Both are hugely important problems to be concerned about.

Fantasies about super intelligent AI revolting is just anthropomorphization - humans revolting (or at least, we used to). The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses.

It’s not necessary to replace democracy if the rich and powerful can bend to the opinions of the populace as it suits them.

However, the mangled headlines make me more likely to click through. Maybe they could leave the “how” and add a footnote: “WARNING: This headline starts with a word known to the State of California to be click-baitish.”

1: https://en.wikipedia.org/wiki/Scunthorpe_problem

and it is weird to call it brave. Posting opinions that loads of other people have on the internet is brave? bravery would be taking an actual stand against someone about one of these things. (granted, there is some bravery to sharing one's writing in public at all, and I don't mean to scorn that, but that's also not the bravery you're talking about.)

Probably there's no use responding to this and getting into a bickery comment thread. I just wanted to express a negative opinion about the format to offset all the positive ones.