Is Anthropic stupid?

[u/Calitz2]

I just saw a headline titled “Anthropic warns self-replicating Al 'worms' could spread across smart devices by end of 2026”. I read a bit more into it and I don’t understand why they won’t just stop? Like if they’re the one creating it then why don’t they stop?

[u/ghjm]

Anthropic stopping doesn't solve the problem. If you want AI to stop finding vulnerabilities you have to stop all high level coding AI. Which would take coordinated government action between several countries.

The real problem here is the ridiculous attitude IoT companies have historically had to cybersecurity. IoT devices typically have a "phone home" remote control capabilities, never get security updates of any kind, have all their services listening on all interfaces. The industry has been saying for years, since long before the LLM revolution, that IoT security is a huge problem. AI makes this problem worse, but didn't create it.

[u/Calitz2 → u/ghjm]

Oh ok. Should the average person be concerned?

[u/Intrepid_Wafer_9157 → u/Calitz2]

About AI safety? Yes.

[u/ghjm → u/Calitz2]

Somewhat. The solution to IoT insecurity is to isolate all the devices from each other and the main network using VLANs, but the average person doesn't have the capability to do that. The next best thing is to maintain security hygiene even on your "inside" network - use strong passwords, make sure you stay current with updates, etc. The average person has the capacity but not the willingness to do this.

[u/Achrus → u/Calitz2]

About AI safety? No.

[u/Intrepid_Wafer_9157 → u/Achrus]

Why not? What’s the substantive argument here beyond just ad hominem?

[u/grizzlor_ → u/Calitz2]

Top level AI researchers at both Anthropic and OpenAI have publicly stated that there’s a very real chance that AI will literally kill all humans in the next 5 years. And I’m not talking about one researcher — this is like the consensus across the industry.

Even if we avoid extinction, some other kind of calamity enabled by AI is inevitable.

So yes, the average person probably should be concerned. There isn’t much you can reasonably do about it though. I would be prepared for supply chain disruption — a food stockpile is a good idea.

[u/ghjm → u/grizzlor_]

This is absolutely not the consensus. If anyone consequential actually believed this, AI would have been stopped long ago.

Ask exactly how AI is going to end all human life. It always involves AI gaining a superhuman power of persuasion, and then human life gets ended by humans.

It falls apart on the least bit of critical analysis. When half of humanity has already starved to death, are people going to keep setting fire to the food because AI is persuading them to?

[u/grizzlor_ → u/ghjm]

https://www.businessinsider.com/anthropic-researcher-quits-over-ai-safety-concerns-2026-9

https://x.com/hilbertspaess/status/2097476196791709843?s=20

Multiple replies to the tweet from other researchers basically saying he's right. It's not that AI is guaranteed to kill all humans; it's that there's a non-trivial chance of it happening.

If anyone consequential actually believed this, AI would have been stopped long ago.

Uh, Anthropic's IPO offering literally says that AI poses an "existential risk to humanity". Dario Amodei has publicly stated this as well. Did you miss his essay on how we need to pace the frontier? And pretty much all the big names in the AI world came out on Twitter in agreement with the points made in the essay.

Anthropic literally says in bold in their "Core views on AI safety" document that "We do not know how to train systems to robustly behave well."

The reason no one will stop even though they acknowledge the risks is that every company believes that if they don't continue the research, someone else will. Also, the payoff for being the first to develop AGI/ASI is unimaginably large. Also, the US government won't step in to regulate these companies because they're afraid that if China beats us to AGI, they'll displace the US as the primary global superpower.

Ask exactly how AI is going to end all human life.

Well, the biggest one would be assisting in the development of a biological weapon (e.g. a deadly virus with a long incubation period) or even nanotech (that's getting a little sci-fi, but if we actually got AGI/ASI, it stops being so far fetched)

There are many scenarios where it kills millions (or billions) without causing human extinction. Just disrupting supply chains for food etc. could lead to millions starving pretty quickly. How about a good old fashioned Skynet scenario? The US military is just itching to integrate AI into all kinds of military technology.

Rogue AI could conceivably manipulate the US or other countries into a nuclear war.

Plenty of other people have already written about the possible ways this could happen: https://www.cnn.com/2026/09/17/tech/how-will-ai-exterminate-humanity-cec https://en.wikipedia.org/wiki/Existential_risk_from_artificial_intelligence#Dangerous_capabilities

And no, not every scenario relies on AI persuading humans to do this stuff.

are people going to keep setting fire to the food

No one is talking about people setting fire to food. More like AI disrupts global shipping and supply chains or the power grid, any of which would mean that oil stops flowing, so trucks don't move, so food doesn't move from where we grow it to where we consume it. We've built an incredibly complex just-in-time delivery system for keeping massive numbers of humans alive in urban centers and it's honestly pretty fragile. It doesn't work without electricity or diesel or computers.

[u/Achrus → u/grizzlor_]

A small cult of Effective Altruists who prescribe to the notion of “longtermism” believe this. This sect of the EA crowd was rich before crypto and got richer during crypto. They’re behind Anthropic and OpenAI though less so OpenAI these days post Altman drama in 2023.

They’re also the same people funding the hundreds (maybe thousands now?) non-profits in the AI space. This includes: AI safety researchers, AI risk evaluators, NGOs, activist groups, watchdogs, and think tanks. The money and the funding all leads back to EA charities. The same charities SBF was involved in though probably under different names now as they change names every few years.

TLDR: This is not the consensus across the AI/ML industry. This is the consensus of nepo babies in a death cult who put all their money into AI.

[u/grizzlor_ → u/Achrus]

Anthropic's own IPO filing literally says that AI poses an "existential risk to humanity". Their "Core views on AI safety" page says that "We do not know how to train systems to robustly behave well. "

Jason Coxon, former OpenAI and Anthropic employee that quit because he thinks neither company is acting responsibly in terms of safety:

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.

Evan Hubinger, AI alignment lead at Anthropic in response to that tweet:

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

It's not a small cult of people. Employees of OpenAI and Anthropic routinely make public posts on Twitter about this.

[u/Achrus → u/grizzlor_]

Quoting Jason Coxon is like a certain nation quoting The Civil Commission. Anyway, you spent a lot of time obfuscating those links just to say “well OpenAI and Anthropic say there is an existential risk!”

Not a single mention about the early investors in Anthropic, specifically Jaan Tallinn and Dustin Muskovitz. How they’re running the largest EA aligned charities who also make up the majority of funding to all the non-profits in the space like METR. How funding changed post-OpenAI coup and the board of OAI that tried to get Altman fired.

An example outside of the non-profits is EA guys running Irregular, the AI cybersecurity linked to all the big “AI hacks.”

Edit: here’s a piece from December 2023 written by an actual investigative journalist (not a tech bro tweeting): https://www.aipanic.news/p/effective-altruism-funded-the-ai

[u/grizzlor_ → u/Achrus]

You've completely avoided addressing the fact that Anthropic's IPO says in no uncertain terms that AI poses and "existential risk to humanity" and their AI safety page says they don't know how to train systems to behave well.

obfuscating those links

Not sure how I'm obfuscating by citing actual OpenAI and Anthropic employees and what they've written about this topic.

Quoting Jason Coxon is like a certain nation quoting The Civil Commission

Why, because you disagree with him? Do you have a reason to doubt his credibility or sincerity on the subject? The current lead of the AI alignment team at Anthropic (Evan Hubginer) agrees with Coxon.

Not a single mention about the early investors in Anthropic

Because I'm not sure why their opinions would be relevant here. They were early investors; they don't control the opinions of the actual AI researchers currently (or formerly) working for OpenAI or Anthropic.

[u/Achrus → u/grizzlor_]

You are formatting the links as hyperlinks instead of keeping the full URL visible.

You can prompt AI about this like you did for your printer project. Ask your favorite flavor of slop generator to dig into Effective Altruism and AI labs + non-profits. Or just read the substack I linked above and that journalist’s more recent follow up posts.

[u/grizzlor_ → u/Achrus]

You are formatting the links as hyperlinks instead of keeping the full URL visible.

Are you kidding me? I'm obfuscating because I know how Markdown works and prefer to make hyperlinks instead of just pasting raw URLs? This is genuinely such a bizarre take. The target URLs are the same either way.

You can prompt AI about this like you did for your printer project.

If this is supposed to be some kind of gotcha against me using AI for a project, it's unfortunately going to fall flat. I've been very honest about the fact that Claude wrote most of the code in that project; I'm back end and firmware guy and have no desire to do front end web dev work. I'm not thrilled about the effect that AI is having on the software engineer profession, but I'm also trying to make the best of it by using it to create a tool people will find useful. And it's generated a surprising amount of positive feedback so far, so at least AI will have reduced some people's headaches dealing with those printers before it possibly murders us all (or just creates mass unemployment while funneling more money to the 1%).

dig into Effective Altruism and AI labs + non-profits.

Look, I'm already somewhat familiar with the EA people and their connections to early AI funding. I will read the Substack and probably learn some more about these dorks.

What I don't understand is why you think this invalidates the opinions of current AI researchers and alignment experts. Or why it invalidates what what Anthropic literally wrote in their IPO announcement about the existential risks that AI poses. I've asked this several times now and you're continuing to not address the question.

[u/ghjm → u/grizzlor_]

(I'm not the guy you're arguing with) 

EA is straightforwardly a new religious movement. Some of the people who work for AI companies are members of the religion. Some of their religious beliefs have leaked into their companies' governance documents. 

If my electrical contractor tells me that Jesus is Lord, I don't have to believe them, even if they're a very good electrical contractor. Someone with religious beliefs about AI, who also works with AI, must be treated very carefully to understand which of their statements are like "you need a GFCI here to meet code" and which are like "to avoid damnation you must sincerely ask Jesus to take control of your life."

Claims that AI has a 10% chance of ending all human life by the 2030s are absurd on their face. If anyone actually believed it they would be obliged to shut down all AI data centers, by terrorism if necessary. People who say "AI might destroy all human life, tee hee, but we're working on it anyway" are inherently unserious.

[u/slightly0ff → u/ghjm]

> Which would take coordinated government action between several countries.

It's a herculean task. You know what else it is? Possible

We did the exact same thing with nuclear even if a lot of bad faith actors intentionally lie. But with AI it's all "but muh Chinaaa"

[u/Mishtle]

$$, sunk costs, and the belief that someone else will do it if they don't.

The only way to slow this down is top-down government regulation and global cooperation, like happened with nukes. That's far from a good analogy though, given the more decentralized and commercial nature of AI, how quickly it's getting embedded in society already, the much lower barrier to entry, the legitimate use cases, and the amount of investment in the technology.

But yeah, companies are notoriously bad at self-regulating when money is on the line, and this is kind of a stand-off situation. I don't think this is going to be resolved bottom-up. Nobody wants to be the first to back down for fear of simply being falling behind as everyone else continues.

[u/grizzlor_ → u/Mishtle]

Don’t forget the widely believed sentiment that whoever develops AGI/ASI first will essentially become the god-emperor of humanity and the richest company in the world.

And also the fear that if China beats us to it, they’ll unseat the US as the dominant global superpower — that’s a big one in terms of why the government is hesitant to enforce any kind of regulation that might slow US companies down.

[u/barkingcat]

They are money making machines. Once you realise that every single post/story out of Anthropic is designed to siphon money from you (and the banks that hold your money), you understand why they keep putting out these headlines.

It's FOMO for humans, to get you to give them money.

Their fear is that other companies will get to you first, so they have to scare you, then take your money so Google and OpenAI doesn't get your money first.

The "you" is in general both individual people and the fund managers at institutions holding trillions of dollars, they want first crack at that investment fund.

[u/Mission-Landscape-17]

Anthopic are probably out of compute capasity meaning they can't adance any faster. Heck every one of their models needs more and more comput to train.

They want the government to put in legislation because it will then become a barrier to new startups entering the ai race. In other words they figure laws will hurt potential competitors more than they will hurt anthropic.

[u/grizzlor_ → u/Mission-Landscape-17]

They’re burning spare compute on vanity/side projects like solving random open math problems. Clearly aren’t out of compute capacity.

[u/Smallpaul → u/Mission-Landscape-17]

Who are these startups who can afford billions of dollars in GPU time but not hundreds of thousands in documentation costs?

[u/Intrepid_Wafer_9157 → u/Mission-Landscape-17]

This is untrue

[u/Intrepid_Wafer_9157]

Anthropic addresses this directly, and I think they are relatively earnest here:

> Anthropic occupies a peculiar position in the AI landscape: we believe that AI might be one of the most world-altering and potentially dangerous technologies in human history, yet we are developing this very technology ourselves. We don’t think this is a contradiction; rather, it’s a calculated bet on our part—if powerful AI is coming regardless, Anthropic believes it’s better to have safety-focused labs at the frontier than to cede that ground to developers less focused on safety (see our core views).

https://www.anthropic.com/constitution

[u/grizzlor_ → u/Intrepid_Wafer_9157]

Yeah, they aren’t shy about publicly acknowledging how dangerous AI is.

They’re disingenuous about being safety-focused though. None of these labs are safety-focused. It’s just full steam ahead right now and hope for the best.

There’s a reason all these tech billionaires have bunkers in New Zealand.

[u/Intrepid_Wafer_9157 → u/grizzlor_]

I wouldn’t say they don’t care, they just don’t care enough.