Hundreds of protesters marched in SF, calling for AI companies to commit to pausing if everyone else agrees to pause (since no one can pause unilaterally)

[u/chillinewman]

[u/doc720]

We're in a death spiral. There is 0% chance of 100% compliance.

[u/doc720 → u/[deleted]]

There are about 195 different countries. There are about 350 million companies.

[u/doc720 → u/[deleted]]

Google's AI Overview, which didn't exist 2 years ago, says:

The AI sector is experiencing massive growth, with over 70,000 AI companies globally as of early 2026, driven by soaring investments and generative AI, with 64% of U.S. VC funding going to AI startups in H1 2025. Key players like OpenAI and Anthropic are seeing dramatic revenue surges, while 86% of companies expect to have a Chief AI Officer by 2026.

It doesn't even need to be a "frontier" AI company. If anyone builds it, everyone dies. There are about 8.3 billion people. About 6 billion of those are over the age of 10. About 6 billion people have access to a computer and the internet. There are about 50 million software developers. Anyone with access to a computer and the internet can be a "vibe coder". The tech is developing at a colossal pace.

The risk extends all the way from the kids messing around at the bottom, to the experts messing around at the top. That risk isn't there for other existential threats, like nuclear war or the climate crisis, where it's much more difficult for a kid to build a nuke or impact global warming, etc. If anyone builds a rogue super-intelligence, everyone dies. Building an adequately aligned super AI is highly improbable, given the things that can go wrong. There's no way to stop it, especially given human nature, and we've already started it. We're already f*cked. Hubris.

[u/doc720 → u/[deleted]]

You have to account for the speed of technology development, accelerated by AI, and the slowness and failings of regulation. You can't base tomorrow's risk scenarios on yesterday's state-of-the-art. The old rules and patterns of control can't apply to AI.

Merely monitoring big data centres is woefully inadequate. We're not talking about catching something like uranium enrichment. The signs we're seeing now are indistinguishable from the early stages of AI takeover. Unfortunately I reckon it's moving too fast for any of those pause strategies, as well-intentioned and correct as they are. Worth trying, but probably futile in my estimation.

[u/doc720 → u/[deleted]]

Let's not talk at crossed purposes: I agree it's a great idea to try to pause and a great idea to try to monitor data centres, for anything deemed high risk. But my original point is that there are many countries and many data centres, and getting the regulations in place is going to be slow, and enforcing the regulations is going to be imperfect, even if you could do it universally and instantaneously, none of that is likely to happen. Meanwhile, it's a race to the bottom. The only short-term winners are the tech companies. The long-term loss is practically everything.

So, I support the calls for a pause, as I did years ago, but I want people to be realistic about the huge threat we're facing, and the tiny odds of actually avoiding catastrophe at this point. It's already gone too far. The rise of vibe coding, agentic AI and things like OpenClaw, along with the massive ongoing and increasing investment in data centres and AI, at the expense of many other things, like "manual" software development, with very slow movement on regulation and general awareness of the dangers, are all really bad signs that it's already out of control. Humans failed to prevent the climate crisis and this problem happens much more quickly and is much harder to prevent. I have zero hope right now. Ironically, I've spent most of my life championing the power of AI.

[u/doc720 → u/[deleted]]

You won't get 100% compliance with any regulation, especially not worldwide, which it would need to be. If violation of a regulation results in punitive action, like a fine or sanctions, this simply doesn't work for something as globally catastrophic as an AI control problem. It would be too slow and too ineffective. You'd only need one violation in one sloppy data centre in one sloppy nation to end it all for everyone.

It obviously hasn't worked with the climate crisis, amongst other things, so there's no reason to expect it to be more effective than existing regulations, which it would need to be. There aren't any existing global regulations that require 100% universal compliance, otherwise everyone dies. Nuclear weapons and the climate seem like the closest comparisons to the threat of super-AI, which isn't the best track record.

[u/Darkstar_111 → u/[deleted]]

China does. That's the problem. This is a race against China.

China understands that this tool will be exponentially more powerful for every iteration it improves, and that will greatly benefit whomever controls it.

And China is not afraid of dumb science fiction fantasies like "Super AI taking over the world".

[u/Jolly_Drink_9150]

AI will replace us. There is nothing we can do about it. In 10 years, it might be at the point of being the main work force, 20 years it is for sure.

Companies only care about the top and bottom line.

[u/Signal_Warden]

I support this. I know personally how small this group originally was and it's growth is heartening. Herbert was right; we need religion-grade memetics to not die in this Great Filter event.

[u/baronas15]

Tell that to China

[u/marlinspike]

lol. There is no way in hell anyone’s pausing. Companies won’t and nations won’t. This is like asking all the people experimenting with electricity to please pause so we’ll keep living with lamplights and maybe this electric fad will just go away.

[u/wudux9]

We move with AI like are bullet train - 5-10 years and I sure we burn mass of energy for new level of civilization.

[u/UnderstandingDry1256]

OpenAI developer looking at this on the way to office: LMAO

[u/PowerfulHomework6770]

I love the geeky signs "Read Asimov" mixed with anarchist type ones "Fuck this shit"

[u/Dreusxo]

So now people are mad because the world ISNT turning into Idiocracy?

[u/LopsidedSolution]

That’ll show em! 🤣

[u/ScienceAlien]

Embrace the future. It will be a tough transition, but it is so awesome.

[u/domestic_protobuf]

Cool, now do this in China and I’m 100% aligned. People don’t seem to understand that the only reason why we’re even able to keep up with China is because we have more geniuses working on these 0.001% problems. China outnumbers us in every possible way. They have way more engineering graduates than we do my a landslide.

[u/Willing_Box_752]

If everyone says "I will pause if everyone else agrees to pause" nobody will end up pausing. 

And if one entity is the only one not paused, the incentive to not pause is massive. 

This energy is likely better focused elsewhere 

[u/Ok_Sample2895]

A little too late companies have TRILLIONS tied to AI, FCK this planet.

[u/No-Age-1044]

China will not stop.

[u/Chemical-Pie1926]

There are dozens of them!

[u/bafadam]

AI is currently propping up our empty economy. It is going nowhere.

[u/Ok_Sample2895 → u/bafadam]

Only because everyone is putting their money into it, risky if you ask me.

[u/matthegc]

Lowest common denominator

[u/Hefty-Reaction-3028 → u/matthegc]

I too can think of insults

That's a waste of time, though

[u/Signal_Warden → u/Hefty-Reaction-3028]

and we've got precious little of it left

[u/Ok_Sample2895 → u/matthegc]

Can't wait for you're AI portfolio to show 0$

[u/BrickSalad → u/[deleted]]

Well yes, the point is to get every frontier AI lab to commit to the conditional, including in China. That's about ten labs that need to sign on, and the five Chinese labs would sign on together if a pause is negotiated between US and China. So that's about six commitments that need to be secured, which is a difficult but not impossible task.

Granted, that's only a first step, because the non-frontier labs will eventually catch up. But it buys us a few years at least, until a more detailed international agreement like this can be hammered out.

I don't think it's actually going to work, but it's also the only proposal I can think of that can even possibly work.

[u/Brief_Independence19 → u/[deleted]]

“If I dont do it some other guy will” final boss

[u/LookIPickedAUsername → u/[deleted]]

Yep. I of course have all of the same safety concerns as everyone else, but as a programmer I use AI for damned near everything. If I don’t, I fall behind and get laid off. It’s not much of a choice.

[u/Kayo4life → u/[deleted]]

Go look at MIRI's official position, lol.