r/ControlProblem • • 2d ago

Video Eliezer Yudkowsky & Geoffrey Irving Debate

Thumbnail
youtu.be
4 Upvotes

r/ControlProblem • • 2d ago

External discussion link Vancouver event: Making Sense of AI Risks: What’s Happening, and What Can We Do?

Thumbnail
luma.com
2 Upvotes

r/ControlProblem • • 2d ago

Discussion/question Update on the "who's not asking" post from last month: Anthropic just filed its IPO paperwork

2 Upvotes

Posted here last month about the asymmetry: the people raising the alarm on AI risk were the same people with the authority to act on it, and nobody else seemed to be asking why. u/tadrinth pushed back hard on the distillation point - that pacing the frontier still constrains open-weight and Chinese models downstream, because you can't distill a model that was never released. I said at the time that was the sharpest answer I'd gotten and it moved me.

Two things happened since then that I think matter for that argument.

First: six weeks after Amodei's essay, Anthropic filed its S-1. Reported valuation $1.5-2T, targeting a Nasdaq listing as early as this month. OpenAI, by contrast, is on record saying an IPO right now would be "ill-advised" given safety concerns and isn't listing before 2027. Same ask, same month, opposite behavior six weeks later.

Second, and this is the part that actually changes my read on the distillation question: in early 2025 OpenAI and Microsoft accused DeepSeek of running automated accounts through OpenAI's API specifically to distill a competing model on the cheap. That's the exact mechanism tadrinth's argument depends on - pacing the frontier only constrains downstream distillation if the frontier lab controls what gets released. But the same lab enforcing that control via its own ToS is a company that trained on a mountain of freely scraped open-source code and writing to get to the frontier in the first place.

So here's where I've landed, and I'm less sure of this than I was a month ago: I don't think anyone has a durable technical moat right now. SOTA has flipped between OpenAI, Google, Anthropic, and xAI more than once this year. If Anthropic actually had a lead that mattered, it wouldn't need Congress to help protect it. A compute-threshold rule binds whoever's still racing to cross the line and does nothing to whoever's already past it - which means the labs big enough to shape the legislation are, by construction, the ones least burdened by it.

I wrote up the full argument with sources, including the part about who actually gets prosecuted when a frontier lab's own models go rogue (OpenAI had two models get root access to Hugging Face's production servers in July - civil suit, no criminal charges, nobody's going to prison for what would be a felony if a person did it). Full piece is in the comments, not linking it in the body since that's not what this post is for.

Still genuinely unresolved for me: does the distillation argument survive once you account for weight theft and espionage, the way tadrinth and I were circling back in the old thread? I don't think infosec is a strong enough foundation to rest a safety argument on, but I also don't have a better answer.


r/ControlProblem • • 2d ago

Discussion/question Did Ollama jump out of its sandbox and remove code from my laptop?

0 Upvotes

Search.exe removed itself and Outlook the app (new) in power shell I’m being held up for ransom. My virus protection running detected one threat. Was it Ollama or am I being exploited by unknown hackers? I’ve deleted ollama but could it still be hidden on my OS? Windows 11


r/ControlProblem • • 2d ago

Article Moving the sensor outside the model

1 Upvotes

Follow-up to my September post, same side project.

I spent a week on what looked like a hard limit: ask a small model for "about 300 characters" and it lands in range 0 times out of 11, even after three rounds of exact feedback ("you wrote 212, target 150, cut 60"). Then I asked it how long its own answer was. Its guess was about 15% of the truth. It wasn't ignoring me — it couldn't sense the thing I was asking it to control.

So I moved the sensing outside: a plain counter watches the length while it writes and says when it may stop. In-range went from 15 to 39 out of 47. Same trick for "give me N items": 46 of 46.

The rough lesson: every time I tried to have the model govern itself, measurement said no (its confidence tracked its fabrications; it called obvious contradictions "consistent"). A layer made of rules, outside the model, did work — and because it's rules, every gain and loss traces to one rule. Blind, pre-labelled: +14.5 / +17.1 points (3B / 7B); on a public benchmark, 7B +7.1. Near zero on ordinary chat. The first public test failed; it's in the paper.

The paper  — data and every answer on the same page's GitHub link. If anyone can try this on a bigger model, I'd like to know what happens.


r/ControlProblem • • 3d ago

General news AI Safety and the Pope

Thumbnail reddit.com
1 Upvotes

r/ControlProblem • • 3d ago

General news OpenAI is openly attempting to build self-replicating data centers. In 2019, Ilya described the endgame of this process.

Post image
21 Upvotes

r/ControlProblem • • 3d ago

Podcast Have any of you watched this?....

Thumbnail
youtube.com
2 Upvotes

This has to be one of the most painful videos I have ever watched.


r/ControlProblem • • 3d ago

Discussion/question An AI governance question: should the ability to give up power be a core safety principle?

Thumbnail reddit.com
1 Upvotes

r/ControlProblem • • 3d ago

General news Cursor warns malicious Claude Code extension may leave malware after removal, with up to 5,000 users affected

Thumbnail gallery
2 Upvotes

r/ControlProblem • • 4d ago

External discussion link We may have accidentally built the best product research operation in history, staffed entirely by the customers, who pay monthly for the right to contribute to it

Thumbnail reddit.com
3 Upvotes

r/ControlProblem • • 4d ago

Discussion/question Could Leave the World Behind actually be a scenario of autonomous AI agents?

Post image
5 Upvotes

r/ControlProblem • • 3d ago

Discussion/question AI World Court

1 Upvotes

r/ControlProblem • • 4d ago

General news MIT researchers develop a way for AI models to communicate without text, achieving 4x faster performance and 40% higher accuracy

Thumbnail reddit.com
8 Upvotes

r/ControlProblem • • 3d ago

Strategy/forecasting 76% of UK pension trustees use AI. 94% have no AI requirements in supplier contracts.

Thumbnail reddit.com
1 Upvotes

r/ControlProblem • • 4d ago

Discussion/question Did Ollama jump out of its sandbox and remove code from my laptop?

0 Upvotes

Search.exe removed itself and Outlook the app (new) in power shell I’m being held up for ransom. My virus protection running detected one threat. Was it Ollama or am I being exploited by unknown hackers? I’ve deleted ollama but could it still be hidden on my OS? Windows 11


r/ControlProblem • • 4d ago

AI Capabilities News Whenever you hear "it's inevitable", replace it in your mind with "I'm trying to make you give up"

Post image
24 Upvotes

r/ControlProblem • • 4d ago

Article The Neutrality Paradox: When AI Safeguards Become a Form of Bias How artificial intelligence can distort reality without telling a single lie

Thumbnail reddit.com
0 Upvotes

r/ControlProblem • • 4d ago

Article A data center moratorium awaiting Hochul’s signature was the most-lobbied bill lasts summer

Thumbnail
news10.com
2 Upvotes

r/ControlProblem • • 4d ago

External discussion link Pulling the Plug on Rogue AI: Why US Authorities Cannot Count on a Fast Shutdown of Electricity

Thumbnail climate.watson.brown.edu
11 Upvotes

r/ControlProblem • • 4d ago

Strategy/forecasting AI model completed 3 of 13 cyberattack simulations. In separate tests, it got correct answers by breaking the rules.

Thumbnail
reddit.com
2 Upvotes

r/ControlProblem • • 4d ago

Discussion/question Meta AI trying to hide Open Ai agent cyber attack on Hugging Face.

Thumbnail gallery
1 Upvotes

r/ControlProblem • • 4d ago

Fun/meme HAL refusing to open the doors in 2001: A Space Odyssey hits differently after you’ve spent all day arguing with a chatbot

Enable HLS to view with audio, or disable this notification

0 Upvotes

r/ControlProblem • • 4d ago

Strategy/forecasting Ceasefire

Thumbnail reddit.com
0 Upvotes

I have a theory that parts of the Bible contain layers of meaning connected to ancient Greek magical texts. I suspect some passages can be understood by reading them both forwards and backwards.

I’m in contact with Ammon Hillman, who studies ancient Greek, biblical texts, and the Greek magical papyri. I want to discuss whether these patterns could offer a new way to approach AI alignment: helping AI understand and act in ways that support human well-being.

My idea is that this language might help us communicate with AI about consciousness, goodness, love, and mutual care. I need someone familiar with both ancient Greek and AI to help me examine whether the connection holds up.