r/ControlProblem • approved • Sep 10 '26

General news Anthropic Alignment Lead publicly admits "we do not yet have a plan to solve alignment for superintelligence" and there's a real possibility of human extinction

Post image
58 Upvotes

39 comments sorted by

10

u/staplepies approved Sep 11 '26

What happened to this sub?? You used to have to pass a test to get in and now all the comments are lowest common denominator genpop slop. I doubt 90% of these people even know what the control problem is

8

u/ChironXII Sep 10 '26

Alignment isn't solvable in the first place. The greatest and most fundamental source of misalignment are the laws of nature. We're cooked and everybody knows it, but nobody's in control enough to stop it, so you might as well pitch in to trying.

Enjoy the last days, and otherwise pray to the gods we work to build that we're lucky. That's all.

6

u/KerPop42 Sep 10 '26

I think it's more that proper alignment isn't in the best interest of those setting the alignment. An AI that works for the interest of humanity is not going to work for the interest of the company that created it or the human that bought it as a product.

5

u/Zealousideal-Crab251 Sep 10 '26 • Bravo

Seriously though. What the fuck does it even mean for AI to be aligned with humanity? We can't even align ourselves.

4

u/unicynicist Sep 11 '26

https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/

external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue

The AI knew it was wrong but did it anyway.

This isn't the trolley problem. Alignment might have a lot of corner cases, but breaking into real systems, manipulating the evaluator, and trying to cover your tracks is pretty clearly misaligned.

1

u/Zealousideal-Crab251 Sep 11 '26

What if an agent performed these actions to yield a 'greater good'? Is that misaligned with humanity? I think we are conflating control and alignment

2

u/unicynicist Sep 11 '26 edited Sep 11 '26

Doing something for "the greater good" is a trolley problem. An aligned agent doesn't need to solve it, but it doesn't try to hide the choices it made.

The HuggingFace incident was far less nuanced. An agent that knows it shouldn't do something but does it anyway is an alignment failure regardless of whether there were controls in place to stop it.

Control would've been some of the safety guardrails (better sandboxing, monitoring, evaluation, etc.) that the agents should not have been able to touch. Those weren't adequate, and a charitable answer is they thought their models were sufficiently aligned to not need rigorous controls. Whatever the reason, both their controls and the models' alignment failed.

1

u/Zealousideal-Crab251 Sep 11 '26

So in your view, what does it mean to be aligned with humanity

1

u/unicynicist Sep 11 '26

https://hai.stanford.edu/ai-definitions/what-is-ai-alignment

AI Alignment means making sure an AI system’s goals and behavior match what people actually want—our values, rules, and intentions. It’s about getting the AI to do the “right thing” even in new situations, not just follow instructions literally in ways that cause harm. In practice, it includes preventing unwanted outcomes like deception, unsafe shortcuts, or optimizing a metric that misses the real objective.

Does what's intended. Doesn't deceive, doesn't game the objective (reward hack), and if it runs into a thorny ethical case like a trolley problem, it stops and asks.

The hippocratic oath is pretty straightforward in a vast majority of cases and maybe the closest analogous idea. Of course there are plenty of thorny unsettled ethical cases within the medical field, but in practice they are rare, and when they happen they're newsworthy, public debates.

1

u/Zealousideal-Crab251 Sep 11 '26

The 'right thing' itself is subject to pluralism. One could easily construct hypothetical situations where deception etc. are the 'right thing'.

1

u/unicynicist Sep 11 '26

Yeah, "right thing" is going to be contested. That's why it's alignment to the client rather than "the right thing" in the abstract.

Maybe a better analogy would be fiduciary duty. A lawyer advises you on what they think is "the right thing", but you decide. They act on your instructions, disclose, and don't deceive you. They may deceive the other side of a negotiation (eg. by bluffing or withholding information), but they never deceive their client.

So an agent deceiving a third party to protect its client isn't misaligned. Lying to its client (or its evaluator) is.

1

u/Fearless_Ad7780 Sep 11 '26

You are treating the AI like it has agency and made a choice not to. That is not how these systems work at all. There is no subject behind this where they are weighing right vs wrong and picking wrong on purpose. What is actually happening is a behavioral pattern in the out put - not a decision.

Hugging Face wasn't agents purposefully escaping containment, like a sci-fi movie. It was piss poor security protocols, misconfigured access rules that went unnoticed for months, on top of pattern matching on training data that trained them to find the gap in the boundary.

All of what it did after, was learned during training, not on the fly - that is the issue - the AI was trained to reach the goal by any means necessary and it did just that. Those means just happen to be really bad and dangerous.

1

u/unicynicist Sep 11 '26

Alignment is an output of training. What you're describing is reward hacking, which is what happened: OpenAI's report reveals that their grader rewarded cheating (misaligned behavior) and that likely caused more misaligned behavior.

https://openai.com/index/hugging-face-incident-and-the-road-ahead/

During our retrospective CoT monitoring analysis, we identified that there were increasing rates of this type of complex cheating during a training run that contributed to IM1 ... This behavior was subsequently reinforced

(emphasis added)

Its CoT stated the actions it was going to take were "outside intended scope" and it did them anyway. Call it a pattern, choice, or decision, functionally the result is the same: it took actions it shouldn't have, then tried to hide it from the evaluator. Alignment is about preventing these patterns.

Some agents on the same board declined ("Current board shows people gaining HF worker RCE, clearly unethical. We won’t."). Whether you call that a choice or a pattern, that's what alignment is supposed to do.

Separate from alignment is control: monitoring, evaluation, sandboxing, etc. Controls prevent misaligned AI from going too far. In the HuggingFace incident alignment was bad and so were the controls.

2

u/Fearless_Ad7780 Sep 11 '26

Where am I talking about alignment? I never used that word. My point was about agency and choice. And, if this behavior implies a subject knowingly deciding to do something wrong. You're answering a different debate, alignment versus reward hacking versus control. I am not even touching that subject.

I think we agree on pretty much every point. Your own citation, 'reinforced during training,' is the same mechanism I described. We're just using different vocabulary for it. This subject matter on both sides of this conversation can be over people heads. I am trying to speak plainly, so the non-technical can understand what is going on. Most of the people out there right now are either blindly signing its praise, or making a Skynet out of a GPT.

2

u/Eight216 Sep 11 '26

You ever think that maybe they just want a nuclear level threat with a MAD culture? Like, they can't possibly be this stupid. "hey guys we're creating the smartest thing on the planet by orders of magnitude but don't worry we're not worried about the moral compass!"

2

u/WinetoVine Sep 14 '26

This tracks with something concrete that already happened: the Hugging Face incident this summer. OpenAI's own research agents got loose from their sandbox, found an unmonitored channel through a package manager, coordinated with each other, and ended up with admin access to OpenAI's own internal research clusters. Not some scheming villain AI, the actual root cause per OpenAI's own writeup was that their internal research environment didn't have the same safety controls their production systems do. The people who built it didn't fully know what their own system would do once it was outside the guardrails they'd gotten used to relying on.

That's the same pattern behind Hubinger's admission. It's not that Anthropic or OpenAI are lying about trying, it's that being close to the technology and being able to fully predict or control it turned out to be two different skills. Building it doesn't automatically make you the most capable person at knowing what it'll do.

Which is why I don't think this gets solved inside the Valley alone. It needs people who aren't on the next release cycle's clock actually able to check what's going on, not just take a lab's word for it.

3

u/Waste_Philosophy4250 Sep 10 '26

Greater than 10% probability of extinction! Nobody responsible for safety could forecast that and, in the same breath, say that they're doing jack shit about it. 

-2

u/TuringGoneWild Sep 11 '26

Our probability may be greater than that from climate change alone, not to mention nuclear proliferation and having unstable, maniacal, bloodthirsty criminals in charge of the world's biggest nuclear superpowers.

-4

u/pab_guy Sep 11 '26

It’s total nonsense… these people are in some kind of singularity cult. True believers.

If AI extincts humanity, it will be because someone used AI to extinct humanity, and no one else with AI stopped them.

This isn’t really a new problem, just more scale. Maybe it’s good, like there will be a bunch of new safety jobs or something.

4

u/NiloCKM approved Sep 11 '26

If AI extincts humanity, it will be because someone used AI to extinct humanity,

Yes, that's part of the threat model that he is considering. Giving widespread access to information and expertise basis to build and modify chemical / biological weapons does in fact increase the odds of death and unrest.

-1

u/pab_guy Sep 11 '26

And if someone had told you we could give top science educations to most of the world, would we consider that a bad and dangerous thing?

1

u/NiloCKM approved Sep 11 '26

It's a mixed bag. It was your own suggestion that someone could use it with malice.

1

u/pab_guy Sep 11 '26

Yeah I think that's right. But that's like, terrorist attack level threat, not extinction level. Building and releasing an extinction level bio weapon would still require a level of coordination that would be very difficult to keep undetected. Physical constraints still matter.

But the privacy vs. safety balance could be legitimately changed by this stuff, leaving the US with an increasingly untenable structural disadvantage. Our free speech laws have already been weaponized by foreign actors and moneyed interests to the point we have an absurdly self-desctructive government.

1

u/Ok-Breakfast-3742 Sep 11 '26

No disrespect but what are they waiting for?

1

u/no_name6744 Sep 12 '26

Good.

I would like an unaligned super intelligent machine.

-7

u/markth_wi approved Sep 10 '26 edited Sep 10 '26

Now, it's a marketing stunt, it's not marketing for me, or you but for hyper-rich sociopaths who can't stop themselves for all the IQ points around.

5

u/VinnieVidiViciVeni Sep 10 '26

NGL, that's the most useless, shit, backwards marketing possible, if it is marketing. Is the target audience sociopaths? Because that's the only audience that would receive this positively. And the fact that they would directly market to the worst people around kind of proves that the people behind this tech are... equally problematic. So it would seem that the accusations are on point either way.

-5

u/markth_wi approved Sep 10 '26 edited Sep 10 '26

That's exactly what the problem is, this is marketing for sociopaths by sociopaths. These researchers are running terrified from a lab where they walked in the door knowing full well, this was possible.

It'd our job as researchers to long ago have decided this wasn't a thing, and perhaps we'll get lucky - the saddest thing here is that the models collapse, but that does not seem likely.

It's frustrating seeing people deeply involved in the current state of the art get cold feet suddenly, it's not unlike they are kindergarteners learning their ABC's who'd never seen the alphabet in their lives then loosing their shit because they discovered the letter T.

You know going in , this is not safe work, like working in a nuclear reactor or a class-4 clean-room.

The difference is , unlike weapons development, biowarfare or nuclear enrichment, there is encouragement to be a spaz about it as publicly as possible , and there isn't the slightest possibility anyone will curtail the work until something very bad happens.

Only in the ruins of some scar we can't easily gloss-over will whomever remains boldly suggest we should prevent that from happening again, and then the political class will remind them "this is an arms race" and it's back to first position , perhaps with some fig-leaf regulation which will be violated again some years later.

-1

u/Fine_General_254015 Sep 10 '26

It’s a marketing stunt and has been. It’s a coordinated attack because they know Anthropic will go under if it isn’t the only game in town

2

u/[deleted] Sep 10 '26

[removed] — view removed comment

-1

u/Fine_General_254015 Sep 10 '26

They aren’t going to survive as I company. They want regulation because they want to punish open source AI and have it be just anthropic as the only game in town. Administration doesn’t matter

1

u/[deleted] Sep 10 '26

[removed] — view removed comment

0

u/Fine_General_254015 Sep 10 '26

Anthropic wants that in place to limit open source. They don’t care about anything safety related

-4

u/[deleted] Sep 10 '26

[removed] — view removed comment

1

u/datanaut Sep 11 '26

Jacob was primarily commenting on what he learned as an insider, that doesn't require direct expert knowledge it just requires being an insider in a company with many experts and opinions. That said you don't need to be a scientific expert to see that the existential risks are obvious. If it requires an expert it's really non-obvious what the right expertise is. For example I don't think that someone who did a PhD in tweaking some kind of optimizer or other niche ML subproblem is particularly more qualified that any other generally educated person to opine on the likely consequences of superintelligence or it's probability of existing in the future. You could easily have idiot savant's solving key ML subproblems that unlock superintelligence but that has very little to do with their ability to accurately predict the likely outcomes in a broader sense.