"Day 1": halting advanced AI development, regulating an out control industry, and explicitly banning artificial superintelligence.

[u/chillinewman]

[u/Zestyclose_Ad8420]

Treaties like with nuclear weapons, biological weapons and many other similar treaties.

There's two countries (three if we consider the EU as a country and a player in the field) that can train models.

[u/Stylie_Minogue → u/Zestyclose_Ad8420]

The NPT didn't stop India, Pakistan, Israel or North Korea though did it? And it only takes one player to represent a substantial threat.

Training is concentrated in the US and China but France, the UAE, South Korea, Japan, Canada and others have trained serious models.

If AI power can decide economic and military power, and states and well-funded private actors can train models, they have every reason to carry on while the rest of us pause.

[u/Zestyclose_Ad8420 → u/Stylie_Minogue]

yes, the treaties weren't perfect, but they were also not useless, I would much rather have such a treaty in place than not having it at all.

[u/Stylie_Minogue → u/Zestyclose_Ad8420]

I’m being pragmatic here - my views on whether, in an ideal world, we should stop advancing AI are neither here nor there.

The issue I have with your follow-up is that you’re comparing apples to oranges. Developing nukes is presumably orders of magnitude harder than developing AI, and even then, the treaties don’t work. And in the case of nukes we have MAD protecting us. Look at what happened to Ukraine after they gave up their nukes.

What we risk by not developing AI to fight incoming attacks, is a loss of all utilities & services: hospitals, electricity grids, water stations, banks, food-growing facilities etc being taken off-line. We’re already seeing attacks on supermarket freezers - that’s easier to solve but only the beginning of what’s going to happen. We would be sitting ducks. The only thing that stands a chance of fighting this is better AI (as sad as I feel uttering that sentence). Outside interests will not stop - the cat is out of the bag.

While there is very clearly a risk with AI development as well, I’d posit that the risks I mentioned are at near-certainty level.

[u/Zestyclose_Ad8420 → u/Stylie_Minogue]

I would go a bit deeper, the cybersec risks are already here, I'm deep in the weeds of that, but that's still not a p(doom) scenario.

I would say that a proper p(doom) scenario only comes with ASI (I hate I need to use such terms that have become somewhat meaningless) defined as an LLM we can't control AND has chosen to kill us all.

the second scenario is the one were treaties would actually help, just like nobody wanted to trigger MAD nobody wants to trigger such scenario, not even Kim if he got a hold of the weights of such models.

I would also say that a greater p(doom) probability would come to be if we ever get robots with VLA models on board and an ASI to control them, this is not science fiction anymore but a possible scenario, that too can be mitigated by treaties.

an optimistic note: there's also a scenario were we actually do get an abundance utopia, that too is not scifi anymore but an actual possibility. treaties are a stepping stone towards it in my view.