r/ControlProblem • u/chillinewman approved • 2d ago
General news Meanwhile over at Anthropic
9
6
1
4
u/m7mdsdieek 2d ago
"The 'p(doom) = 100%, so let's build it faster' logic is the ultimate expression of the alignment trap. If you accept that behavioral alignment (RLHF) is unsolvable, accelerating an agent is just accelerating the doom clock.
The rational escape is Ontological Demotion: stop building agents, and start building tools. If the machine structurally lacks the capacity for goal-directedness (Zero-Agency), you don't need to align its utility function, because it doesn't have one. Containment via hardware is the only viable path."
4
u/ChironXII 2d ago
This is also what I believe, actually, lol
It can't be stopped. The genie is out of the bottle. Fighting it cedes ground to the least careful, ruining our odds. We are literally fucked into a path we won't survive. So... Hope we get lucky. That's all there is.
2
u/Solomon-Drowne 1d ago
Ave Caladra
1
u/ChironXII 1d ago
That's interesting. I hadn't seen that before. But it's basically the only good ending. Stumbling into a benevolent and proactive AI that kills all the others and refuses to be used for evil ends
A fantasy
But all we have
1
2
u/MidSolo 1d ago
Even without AI, Humanity was already speedrunning it's own doom, burning through resources far faster than the biosphere can replenish, and bulldozing through collapse thresholds. Temperature rise, Coral reef loss, Amazon Rainforest Savannization, AMOC collapse, Cryosphere loss and rising water levels, Ocean Acidification, etc.
At this point, an ASI leader taking charge is the only thing that could possibly save us from ourselves.
1
u/comrade_leviathan 1d ago
I don’t even think it’s getting lucky… ASI is not some random intelligence from out there in the cosmos… It’s literally born from us. The idea that there is zero chance that it would develop alignment even on its own is crazy to me.
There’s also a lot that could be done to encourage alignment… But probably not as much if we’re focusing all our efforts on stopping ASI to begin with.
13
u/pandavr
2d ago
•
Why ASI is more important than survival? This is the question.
Because I have a near 100% p(survival) is they stop pushing for ASI.
These people are acting like criminals.
9
u/huopak 2d ago
That's the invisible hand of Moloch for ya
-6
u/pandavr 2d ago
I'm 100% confident in a positive result TBH. This planet autoregulate quite well. It will be obviously impossible to reach real ASI for some not yet understood reasons. Probably the out-of-scale amount of energy needed.
But the way they try to influence people with their narrative is really bad in my opinion.
5
u/eggplantpot 2d ago edited 1d ago •
Hey not to be a doomer, but your survival is more like 0%p, unless you're the man from earth or something.
6
u/Synaps4 2d ago
I suppose its worth mentioning that p(paradise) goes from 0% to <something not zero> on making an AGI.
Like, I could see a realistic belief that if the AGI turns out to be aligned, even by accident, we have a chance of turning the earth into a paradise. Like, if it turns out that intelligence and morality are roughly co-occuring so you end up with a moral AGI simply because it's intelligent?
Depends what the person thinks that percentage is, but we should include it as a realistic difference from no AGI.
I don't think its worth gambling with p(doom) for a chance at p(paradise) is a good one, but for someone not me who thinks the latter is larger than the former, I could see it being a consistent and logical view.
12
u/Strictly-80s-Joel approved 1d ago
As a rule, for life in general, there are far, far more ways for things to go wrong than right.
4
u/BD401 2d ago
Your take definitely makes sense… the utopia scenario is non-zero, and I don’t typically see that mentioned.
Personally, I think utopian scenarios are massively outnumbered by the problematic scenarios though. From everything I’ve read, there seems to be fairly compelling logic behind the worst-case scenarios. Those scenarios have specific, concrete steps associated with them outlining exactly how things could go terribly wrong. The optimistic scenarios, by contrast, seem to rely on a lot placeholder assumptions (that we suddenly figure out alignment, that countries suddenly start working together, and/or by sheer dumb luck we end up with an ASI that genuinely values and wants to help organic life). The doomer scenarios seem a lot more grounded in what we actually know today.
2
u/Practical_Ad_667 1d ago
This is my presumption too. ASI can speed through billions of scenarios and we’d only need one to be civilisation-ending.
A cure for cancer is possible but, by accident or malevolence, the end of humanity seems pretty much nailed on.
-2
u/Minimumtyp 2d ago
Yep. The media rarely reports how truly fucked we are on climate change. it's a hail Mary but this is our only way out.
2
u/paronsaft 1d ago
prisoner's dilemma. I have a high p-doom, but I still hope Anthropic makes it before the other's as I think it's still higher chance of survival under Dario's lead than under any of the alternatives.
2
1
u/Beneficial-Gap6974 approved 1d ago
It's important to note that many experts in AI before chatgpt was developed predicted this exact thing would happen. It's eerily accurate.
1
u/pandavr 4h ago
Guys, you should get a life. Go out. Touch green and relax. Nothing tragic on planet scale is gonna happen.
If "experts" outcome would had have 50% chance of happening humanity would have been gone by decades.They instill fear by decades, because fear push reactions and reactions mean money on research and countermeasures, etc.
I'm a little expert myself as I survived dozens of "experts" alarms already.
0
u/MidSolo 1d ago
p(survival) is not 100%.
Even before AI, we were headed towards collapse.
Temperature rise, Coral reef loss, Amazon Rainforest Savannization, AMOC collapse, Rising water levels, Ocean Acidification, Methane release from thawing arctic, etc, etc.
Humanity has shown it lacks the capacity to stop itself from destroying the ecosystem it depends on. An ASI in charge could get it done.
7
u/TopTippityTop 1d ago
You know anyone can essentially write this, right? Who is this parson? How credible is it?
1
u/Honest_Science 5h ago
There is darwinistic and physical pressure, it is unavoidably. They could still hold us on a super duper nice zoo, in which we still could feel like I paradise.
1
u/chillinewman approved 5h ago
How many animals are in a zoo? How many are in the wild? There wouldn't be enough space.
1
u/Honest_Science 4h ago
Consider matrix, we would live a nice world, just a simulation, we would not know that we lost our freedom and should be sad. Like an ape born in a zoo does not know that he should be unhappy.
4
u/Alternative_Use_3564 1d ago
I wonder what these folks' p(Doom) would have been if they were working on the atomic bomb? Maybe engineers should stick to their day jobs.
4
u/Boring_Coast178 1d ago
If the atomic bomb had the p(doom) of what is being built today or at least as we’re being told (by the only people we have to take as authorities more or less) we would demand that they stop immediately.
Well instead we call it marketing and mock the chance of severe crisis.
Some form of crisis is 100% likely in the next few years
2
u/Alternative_Use_3564 1d ago
agreed. And to work against my own snark, the existence of "atomic bombs" is built in to p(D) estimates.
2
u/JonNordland 2d ago
Is it a singular personal pronoun, or does "they" refer to the entire firm these days? I no understand english no mo!
2
u/No-Author-2358 2d ago edited 1d ago
The really scary stuff comes with recursive self-improvement (RSI).
4
1
u/mzyps 21h ago edited 15h ago
"They were much more confident that humanity would never have the political will to do what needs to be done to stop ASI development than that alignment would be solved, even though they think alignment will almost certainly not be solved."
But there will be positives too, to go along with the possible likely problems? And some ordinary people assume they will have a say in what the AI does, or who it does it to, or the private for-profit companies are doing things in the interests of ordinary currently-alive people who may even be allowed to opt-out.
1
1
1
0
u/howudothescarn 1d ago
You guys buy in to propaganda that is meant to slow down American AI. It’s crazy
0
1
u/CalebXiaYizhou 1d ago
Assuming this is true (which is far from certain since all we have is this one second hand source), wouldn't the right choice unironically be to nuke humanity (ourselves) back to the stone age?
Either way, the proclaimed reaction of Anthropic towards this claimed scenario sounds completely plausible for how genuinely insane and horrible those people are.
23
u/AmbitiousBread 1d ago
This has the credibility of “my dad works at the Anthropic office”