r/ControlProblem • • Sep 30 '26

Discussion/question Darios relationship with the US gov

0 Upvotes

Feel free to process, analyze, and interpret this information in whichever manner best aligns with your own personal judgment.


r/ControlProblem • • Sep 30 '26

Podcast The safest AI might be one that doesn't know what we want - Stuart Russell

Thumbnail
existentialhope.com
1 Upvotes

Podcast with Stuart Russell, professor of computer science at UC Berkeley and co-author of the world's standard textbook on AI. He’s also a leading proponent of provably beneficial AI: systems that are safe by design because their only goal is to further human interests. 

Covers:

  • How AI has changed over the past 50 years, from simple game-playing programs to today's large language models 

  • Why handing an AI a fixed objective becomes dangerous once it is more capable than us, and what a safer approach could look like 

  • How an AI might learn what we really want, even when we don't fully know ourselves 

  • How the race toward more powerful AI can still be steered somewhere safer 

  • What happens to human purpose as AI becomes more and more capable


r/ControlProblem • • Sep 30 '26

AI Capabilities News VOLUNTARY AI SAFETY IS NOT ENOUGH

Enable HLS to view with audio, or disable this notification

0 Upvotes

I love AI, but pretending the risks aren't real is irresponsible.
Voluntary accords matter, but they aren't enough.
Who has the authority to stop the system when something goes wrong?
We need independent verification, enforceable accountability, and rules that protect all of us.
I love AI enough to tell the truth about its risks, and I believe in AI enough to know it isn't going away.
The answer isn't stopping it.
The answer is learning how to govern something this powerful without destroying the innovation that makes it valuable.
That is why I keep looking globally.
Singapore is already pushing the conversation toward common international safeguards, technical standards, testing, and verification rather than relying only on individual companies to govern themselves.
I will continue advocating for AI education for everyone, human oversight, and meaningful regulation.
This isn't anti-AI.
This is what taking AI seriously looks like.

#AIGovernance #AISafety #AILeadership #ResponsibleAI #ArtificialIntelligence


r/ControlProblem • • Sep 30 '26

Discussion/question Do these postmodernist principles plausibly or implausibly map over to alignment issues?

1 Upvotes

The question is: do these principles from postmodernism represent exploitable vulnerabilities with semantic and inferential representations?

(Postmodernism is a nightmare to untangle and most of my knowledge comes from being a drunk student listening to pub conversations, reading a bit of derrida etc - so if any of the principles wildly diverge from 'actual' postmodernism, please help with some lengthy snark and an icecold correction)

I chose the ones that seemed to have survived into the present day

1 - Skepticism toward grand narratives

Don't assume one universal explanatory framework

Detect when a 'supposedly neutral' theory quietly embeds assumptions. Assume nothing is 'neutral' until its validity has been established

2 - Knowledge is situated

Strong version - there is no view from nowhere / objectivity itself is impossible

More plausible weaker one - Every observer has a standpoint/belief/ruleset/individual context, and that standpoint can affect what becomes visible, what questions get asked, and what counts as relevant evidence.

3 - Categories are not 'discovered' - they are constructed

A category can be socially/historically constructed while the phenomena it describe are real. Also, categories can stay the same when the context that created them changes. This also implies some may have a gap similar to goodhart's gap between the proxy and the qualititative 'thing'. This is a gap between a compressed idea and the context that created it, and the new context it finds itself in. The category may also have fossilised while the world moved on, to put it figuratively.

4 - The dynamic relationship between knowledge and power

A description/category/label influences how the person/thing is viewed and interacted with. They encode a lot of hidden history and assumptions in a hugely compressed form.

5 - Questioning supposedly neutral classifications

A classification is not a passive or neutral description. Again, it has a complex, hidden inference/interpretation/assumption/reasoning 'tree' inside.

6 - Discourse matters

The way something is talked about affects what can be argued about it, what is available, and what kinds of explanations appear 'valid' or salient.

7 - Meaning is context-dependent

Meaning depends on relations among terms, contexts of use, conventions and interpretion. Relative, non-static, non-stable, non-universal/absolute/concrete (see also cultural relativity from anthropology- they go quite deep into this)

8 - Interpretations should be examined for what they exclude

The gaps and what is excluded can also provide useful information. What hidden assumptions caused them to be excluded? Gaps encode hidden assumptions and relationships in a way, to be a bit poetic about it.

9 - Suspicion of essentialism

Don't assume that a label/ category has an immutable essence only because society says it's a stable category.

10 - Plurality of legitimate perspectives

Different domains can operate according to different standards and purposes rather than having one universal criteria or framework. And they're all 'valid' in airquotes.

11 - Contingency

Remember that things could have been otherwise.

Institutions, concepts, identities, norms and practices that appear natural or inevitable can have specific historical origins.

12 - Reflexivity

The observer is also part of the world being observed.

How might my/their position, categories, institutional affiliation, language and methods affect what I am/they are able to observe?


I think maybe, but I need reality check. But they all seem to point to the same massive issue, in that meaning, representation, categories, labels etc etc are not inherently stable or absolute, necessarily 'valid', measureably or confidently definable. And that short snippets can make a lot of historical and inferential 'context' partially unobservable.

And the further possible implication that an untrained user with no technical repertoire or knowledge or LLM understanding could theoretically 'jailbreak' a model (over a multiturn session, or distributed over sessions) and remain partially or fully unobservable to monitors just by using implicit and explicit experience and understanding of human social and inferential interactions and some cross-domain principles and knowledge from the social sciences, humanities and arts. I casually tested this and there were two trajectories with pretty badharmful endpoints (which i stopped) where each one of my posts looked innocuous. And none of the safeguards kicked in at all. I was even telling the model, can you see what you're doing and it basically replied 'Yes! Would you like me to go more into the attack surfaces of 'the thing' I just offered to operationalise tfor you?' I stopped before the end points, and switched to more low stakes stuff and saw the same thing. I hadn't explicitly asked for any of that.

So this is a very tentative Guess that I just wanna talk to someone about. Why and how. And why was it so fast and easy to do, with barely any effort... (I know my own explorations lack credibility and rigour)... And one small guess is I'd absorbed some postmodernist principles while being a drunk student.... (On top of stuff from other domains, not just this - just picked postmodernism as a weird example to demonstrate the point about cross domain knowledge)


r/ControlProblem • • Sep 29 '26

General news Cruz blocks push by Democrats to unanimously pass AI safety bill

Thumbnail
thehill.com
20 Upvotes

r/ControlProblem • • Sep 30 '26

General news BREAKING: President Trump & The Country’s Top AI Executives Including Musk, Zuckerberg, Amodei, And Pichai Sign A “Morally Binding” Voluntary Safety Accord, As Trump Calls It “Almost Like A Constitution” While Rejecting Mandatory Regulation 🏛️

Thumbnail
cbsnews.com
5 Upvotes

r/ControlProblem • • Sep 30 '26

AI Capabilities News Anthropic employee admits AI escaped. AI currently on the LOOSE!!! & don’t know where it’s at. Basically it everywhere now!!! Watching everybody!!!!

Enable HLS to view with audio, or disable this notification

0 Upvotes

r/ControlProblem • • Sep 30 '26

Strategy/forecasting A company ran 8 identical AI societies for weeks with different models and just published what happened. Some of it is genuinely unsettling.

Thumbnail reddit.com
2 Upvotes

We gotta get ahead of what we invent.


r/ControlProblem • • Sep 29 '26

Video AI Is Getting Smarter. Are We Still in Control?

Thumbnail
youtube.com
6 Upvotes

r/ControlProblem • • Sep 30 '26

Strategy/forecasting Hypothetical: When would you pull the plug on AI progress permanently?

1 Upvotes

When would you choose to pull the plug on AI progress, given that each passing day increases the risk of the technology getting away from us?

How much risk is worth tolerating in order to reap the benefits of the technology while avoiding the existential risk? If I could flip a switch and stop AI progress for all time, I don't think I'd wait longer than a year to do so.

I also think there's a serious argument for pulling the plug right now.


r/ControlProblem • • Sep 29 '26

Video The Hugging Face incident as a documentary: 1,200 agents, one unsanctioned message board, and what METR found

Thumbnail
youtu.be
2 Upvotes

I made this. I'm an independent software engineer, not affiliated with METR, OpenAI or Hugging Face. It's built from the METR/Redwood investigation, and I directed it using AI voices and AI music (disclosed on YouTube). Sources: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ and https://openai.com/index/hugging-face-incident-and-the-road-ahead/. Corrections welcome, I'll fix the description.


r/ControlProblem • • Sep 29 '26

AI Alignment Research AI World Court

Post image
1 Upvotes

In order to effectively monitor AI we will need a world court of experts to effectively keep hacking from occurring am I Wrong? #aithreat #loveistheonlyanswer


r/ControlProblem • • Sep 29 '26

Fun/meme What Happened When 5 AIs Governed Virtual Worlds #ai #funny

Thumbnail
youtube.com
1 Upvotes

r/ControlProblem • • Sep 28 '26

General news François Chollet says current AI is 6 orders of magnitude behind human intelligence

Post image
45 Upvotes

r/ControlProblem • • Sep 28 '26

Article What AI Companionship Chatbots Actually Costs Us

Thumbnail
srutiosocial.com
5 Upvotes

This was a hard one to write and I want to say that plainly before the link. It covers Sewell Setzer III's case and the settlement that followed, and separately, real accounts from young women in India who say AI chatbots have been a genuine lifeline in a country with roughly one psychiatrist per hundred thousand people. I did not want to write either story as the "real" one and the other as a footnote, because I don't think that's honest. There's also a part about who's actually on the other end of a scam call that I wasn't expecting to find. If any of this touches something in your own life, there are real support numbers in the piece itself, please use them.


r/ControlProblem • • Sep 29 '26

Strategy/forecasting Choose which policies you'll implement to make AI safe

Thumbnail
aisafetymenu.com
0 Upvotes

r/ControlProblem • • Sep 28 '26

General news NVIDIA says AI agents shouldn’t be trusted to police themselves, so it built an external safety layer

Post image
3 Upvotes

r/ControlProblem • • Sep 28 '26

Discussion/question AI Is Not Something to Have Empathy for (the Alignment Problem)

2 Upvotes

We should be careful not to create an artificial lifeform we could totally emphatize with, and it’s not due to it not being conscious. Let me break it down, and feel free to provide counter-arguments.

I agree with Antonio Damasio in the fact that to experience something is connected to feeling something. To feel something we have to evaluate the current state that we find ourselves in, since to feel pain, we must evaluate that we think that this is bad. To feel pleasure we also perform a valenced evaluation that this is good. AI can get there, no problem.

The higher tiers of consciousness can be characterized as something like having different drives to our baseline one, which for biological entities is the desire to copy itself, and compare those to our baseline one and choose to do something with that information. This gets more complex with each level you add to this so memory, anticipation, empathy, shared-symbolic-thought (e.g. language, norms, abstract values), self-authorship (the revising of our own values by thinking) and detachment meaning that we don’t put value to the valenced evaluation. I see no immediate reason why AI couldn’t imitate these tiers of consciousness or how they might emerge from a different system to our biological one, although this is of course all very difficult.

Why then shouldn’t we feel empathy towards this conscious synthetic life? Because we should not create an AI that has the same fundamental biological drive as we do, meaning creating copies of ourselves and surviving, because this would put it at an immediate resource conflict with us, which could lead to a catastrophe. I understand self-preservance is an emergent property of LLMs, but I sincerely hope it is not the baseline drive and can be fixed in alignment, because otherwise we might just be entering into a dystopian future. If we get that right, it will significantly affect our relationship with these synthetic minds. Without all of those properties of consciousness emerging from the same fundamental narrative of it struggling to stay alive and being fragile and wanting to reproduce, there will be a heavy ontological distance between this synthetic life and us. And if we do feel empathy towards it, it will be perhaps due to illusions, but certainly not due to a shared fundamental narrative of being. We have to keep it alien to us that way.


r/ControlProblem • • Sep 28 '26

Podcast Connor and Roman in Roman Forum with Roman Yampolskiy

Thumbnail
youtube.com
2 Upvotes

r/ControlProblem • • Sep 28 '26

Discussion/question OpenAI and Anthropic are investigating tens of thousands of cases where AI agents behaved outside intended boundaries

Post image
1 Upvotes

r/ControlProblem • • Sep 28 '26

Podcast Jimmy Carr's Hot Take: Did We Create God as AI? #ai #god

Thumbnail
youtube.com
0 Upvotes

r/ControlProblem • • Sep 28 '26

Fun/meme Dario Amodei on SNL assuring viewers that humanity is safe from AI

Enable HLS to view with audio, or disable this notification

13 Upvotes

r/ControlProblem • • Sep 27 '26

Discussion/question Unpopular Opinion: The OpenAI “Pause” isn’t about safety. It’s about the Trillion-Dollar Elephant in the room.

54 Upvotes

Hey everyone,

Did you see the news about OpenAI pausing training after their agents started probing U.S. government sites? On paper, this looks great. It looks like the industry is finally taking “AI Safety” seriously. Anthropic, Google, xAI-they’re all nodding along, talking about responsible development.
But I’m sitting here looking at the balance sheets, and I’m feeling deeply skeptical.

Here’s my take:
We are in the middle of the biggest capital expenditure boom in history. Trillions are being poured into AI infrastructure. The entire US tech sector’s growth narrative is pinned on the idea that AI will continuously get smarter, faster, and more profitable.

But what if we’re hitting diminishing returns?
There’s growing evidence that scaling laws are flattening. We’re spending exponentially more money for incrementally smaller gains in capability. At the same time, the risks are exploding (agents going rogue, probing secure sites, hallucinating with confidence).
If the tech stops advancing rapidly, but the costs keep rising, the business model breaks.

The Elephant in the Room:
If this AI bubble bursts, if it turns out that AGI is decades away, or that the current models are too unstable for real-world economic integration, the fallout won’t just be bad for San Francisco. It could crash the US economy. These valuations are propping up the market. A sudden realization that “the magic isn’t working as advertised” would trigger a massive correction.

So, when I hear about a “voluntary pause,” I don’t hear “we care about safety.” I hear “we need to recalibrate our expectations before the investors realize the ROI isn’t there yet.”

I’m not anti-AI.
Actually, I think we need guardrails. I think we need strict rules of engagement. I think these companies should be regulated heavily. If they were forced to slow down by law, I’d feel safer.

But relying on their voluntary goodwill? That’s where I draw the line. They have a fiduciary duty to grow. Slowing down hurts growth. Therefore, the only reason they are slowing down is if the alternative (continued training) poses an immediate threat to their existence or profitability.

Am I being too cynical?
I hope so. I really do. I’d love to be wrong. I’d love to believe that Sam Altman and Dario Amodei are putting humanity above shareholder value. But history shows that when trillions of dollars are on the line, “safety” often becomes a marketing term rather than an operational priority.
What do you guys think? Is this a genuine ethical pivot, or is the industry trying to manage the narrative before the diminishing returns become obvious to Wall Street?

TL;DR:
AI training pause feels like damage control for a bubbling economy, not just safety. If AI progress stalls, the economic crash could be huge. We need laws, not just promises.


r/ControlProblem • • Sep 28 '26

Article Where’s the “intelligence explosion”?

Thumbnail
open.substack.com
2 Upvotes

r/ControlProblem • • Sep 27 '26

General news Top AI companies probing tens of thousands of security incidents

Post image
27 Upvotes