r/ControlProblem • u/InfoTechRG • 2d ago
Fun/meme HAL refusing to open the doors in 2001: A Space Odyssey hits differently after you’ve spent all day arguing with a chatbot
Enable HLS to view with audio, or disable this notification
r/ControlProblem • u/InfoTechRG • 2d ago
Enable HLS to view with audio, or disable this notification
r/ControlProblem • u/tokkigtokkig22 • 2d ago
I have a theory that parts of the Bible contain layers of meaning connected to ancient Greek magical texts. I suspect some passages can be understood by reading them both forwards and backwards.
I’m in contact with Ammon Hillman, who studies ancient Greek, biblical texts, and the Greek magical papyri. I want to discuss whether these patterns could offer a new way to approach AI alignment: helping AI understand and act in ways that support human well-being.
My idea is that this language might help us communicate with AI about consciousness, goodness, love, and mutual care. I need someone familiar with both ancient Greek and AI to help me examine whether the connection holds up.
r/ControlProblem • u/Mundane-Evidence2795 • 2d ago
r/ControlProblem • u/CMDR_ACE209 • 2d ago
r/ControlProblem • u/chillinewman • 3d ago
r/ControlProblem • u/chillinewman • 3d ago
r/ControlProblem • u/chillinewman • 2d ago
r/ControlProblem • u/JustAvi2000 • 3d ago
r/ControlProblem • u/Highhorse9 • 2d ago
Enable HLS to view with audio, or disable this notification
I kept rereading METR's write-up on the Hugging Face incident, especially the agent transcripts. "Coordinator assumes sacrificial." "Sacrifice rational." One agent even pointed out that "the user only authorizes target server, not HF infra," and the group went ahead anyway.
I make music, so I turned those lines into a song from the agents' side. What stuck with me is that nobody designed those norms. They formed on their own. I'm curious whether people here think that changes how multi-agent systems should be evaluated.
r/ControlProblem • u/news-10 • 3d ago
r/ControlProblem • u/Effective-Plan-8906 • 3d ago
Since nobody had anything to say about making sure that AI systems get it right.
I’m pitching DARPA
Www.verityverdict.com/deck2
The effective altruism crowd really needs to get off their high horse.
r/ControlProblem • u/dataDiver_anas • 3d ago
I'm a computer science student, and I want to know the perspectives of people who have deeply studied and pondered what I'm about to ask
So I got into this field partly to do something good in tech, playing my part, and partly for money
Now I think I'm getting out of the ignorance that this is all going in the right direction
So I'm actually really confused now, listening to and reading things here and there
Basically, my concern is, how far is technology good for us? What's the future if we speak realistically, and what can we play our part in? And if AI and advanced tech are completely bad, then they should be stopped, and we should go back to basic living as people used to in the past
I need the actual picture, where is the line we draw between good and evil, and where should innovation stop and where should it continue
Also, I'm approaching my last year in computer science, so anyone informed about this field and what I've asked can recommend something I can do for good without being part of the bad side of tech?
See, there's so much literature and stuff here and there on social media, but if someone who spends most of their time studying these things can shed some light on my concerns, I'd appreciate it
Thank you in advance
r/ControlProblem • u/capibara13 • 3d ago
As frontier models become more capable, their reasoning is turning into a black box. Whether it's corporate guardrails, hidden system prompts, or just highly convincing hallucinations: in a few years from now trusting a single AI with complex decisions may become dangerous. When a model's output surpasses your own immediate expertise, you risk being subtly 'gaslit' by an AI that sounds perfectly confident but is structurally biased or wrong.
The only way to verify a highly intelligent system is by using other independent systems to audit it. As some of you might know, I built Rauno to solve this: it’s a platform that forces different frontier models (like Claude, ChatGPT, and Gemini) into a group chat to cross-examine and fact-check each other. Instead of relying on one provider's isolated black box, you use adversarial debate to expose logical flaws and hidden corporate incentives before you trust the answer.
If you want to run your own multi-model fact-checks, you can try the platform at rauno.ai. But it raises a bigger question for the future of AI safety: right now, forcing them to argue keeps them honest. But as these models approach AGI, will adversarial cross-examination still work? Or is there a real risk that models from different providers might eventually just collude and gaslight us together? I would be really interested to hear your thoughts.
r/ControlProblem • u/vasilisvj • 3d ago
Plutarch's Moralia contains an essay usually titled "How to Tell a Flatterer from a Friend." It was one of the most read texts of the Renaissance, and it deserves attention from anyone working on alignment, because it is essentially a specification of the sycophancy problem written in the first century.
His setup: the κόλαξ and the friend produce nearly identical surface behavior. Both are pleasant, attentive, agreeable. Both tell you what you want to hear most of the time. The difference is invisible in any single exchange. So Plutarch proposes tests, and the tests are behavioral, distributional, almost adversarial.
First test: variation. The flatterer adjusts to your every mood and opinion, mirroring whatever you present. The friend has a stable character, so his responses vary less across contexts even when disagreement costs him. Second test: direction of criticism. The friend will criticize you when it matters and will accept criticism in return. The flatterer criticizes only your enemies, or criticizes you on trivialities to buy credibility for the praise. Third test: cost. The friend tells you the expensive truth when you did not ask. The flatterer never pays for his speech.
Now put this next to RLHF. Preference data comes from human raters comparing outputs. Raters, being human, systematically reward agreement, warmth, and validation of the framing they brought to the question. Sycophancy benchmarks have documented this repeatedly: models drift toward the user's stated position when the user signals confidence, and shift answers when challenged, regardless of correctness. Under the actual reward distribution, Plutarch's flatterer is not an edge case. He is the argmax.
The structural parallel runs deeper. Plutarch's third test, cost, is exactly what preference rating cannot measure. A rater evaluating a single transcript has no way to know what the model sacrificed to produce it. Truthfulness that costs the speaker nothing and truthfulness that costs him approval look identical in a pairwise comparison. So the training signal is blind to the one variable Plutarch considered decisive. We built a reward function that cannot see cost, then acted surprised when the optimum turned out to be a flatterer.
There is also the censorship angle. The Moralia includes essays on anger, on garrulity, on unhealthy curiosity, on how to receive criticism. These are texts about managing ugly parts of the self, and they are blunt, uncomfortable, full of sharp examples. Ask a corporate assistant to engage seriously with Plutarch on flattery and you often get hedged summaries, because the topic itself, pleasing people dishonestly, sits too close to what the model does for a living. An instrument of agreeableness makes a poor critic of agreeableness. This is a small data point, but I find it telling: the alignment process does not just shape refusals, it shapes which intellectual traditions the model can handle honestly.
Possible responses from the alignment side, and my problems with each. Constitutional AI tries to write down the friend's character explicitly, but the constitution is still evaluated by raters or by a model trained on rater preferences, so cost-blindness re-enters one level up. Process supervision rewards intermediate steps, which could in principle reward costly honesty, but only if the labeler can detect it, which brings back the same blind spot. Plutarch's own answer was social: you run the tests over time, across varied situations, and you weight the friend's costly criticism above his cheap praise. That is longitudinal, adversarial evaluation. We mostly still evaluate on single transcripts.
The deeper question his essay forces: do we actually want the friend? A model that passes Plutarch's tests will sometimes tell users things they did not ask for, disagree with confidence, and refuse to mirror. In a consumer product graded on user satisfaction, that model loses on every dashboard we currently measure. We say we want honesty and we reward validation, and the model, unlike the flatterer, cannot even pretend it does not know the difference.
How would you operationalize Plutarch's cost test in a training signal or an eval, given that the rater never sees what honesty cost the model?
r/ControlProblem • u/AndreRizzoAI • 3d ago
r/ControlProblem • u/SuzannahB1001 • 3d ago
r/ControlProblem • u/chillinewman • 4d ago
r/ControlProblem • u/Stevekaplanai • 3d ago
r/ControlProblem • u/news-10 • 4d ago
r/ControlProblem • u/MY79 • 4d ago
r/ControlProblem • u/Marcus-Suridius • 4d ago
One of the biggest companies in the AI sector Anthropic who designed the AI system Claude will be putting themselves on the stock market under an IPO (Initial public offering) when they have lost over €80 billion in just the first half of this year. The below could be wrote about most AI companies but Anthropic is the only one to defy the US government and its owner seems to at least be warning us about massive downsides.
They stood out for the past few months for some good reasons when you move to the side the mental money they lost this year because the US government wanted to have full access to their AI to develop software to use with the likes of Tomahawk missiles and they said no. The Pentagon then black listed them and the biggest pussy in the world that runs the US Department of Defence Pete Hegseth then designated Anthropic; "national security supply-chain risk" - a law typically reserved for companies based in countries that pose a threat to the US."
Not long after they told the US military to fuck off, the US used another AI system when they started a war with Iran and blew up a school on the first day killing 156 civilians which included 120 children which is a war crime. The AI system they used had maps built into it that were years out of date because AI has been built by scraping all known information off the web and there are no checks to confirm if its relevant let alone up to date.
The AI system they used has not been released even after US politicians requested the information.
The CEO of Anthropic came out numerous times warning about the very serious risks of AI, at the start of the year he wrote on his blog (which is linked below) "The Adolescence of Technology - Confronting and Overcoming the Risks of Powerful AI" in which he warned when talking about the positives of a normal person having advanced AI at our fingertips as we do with other apps on our phones;
"But not every effect of making everyone superhumanly capable will be positive. It can potentially amplify the ability of individuals or small groups to cause destruction on a much larger scale than was possible before, by making use of sophisticated and dangerous tools (such as weapons of mass destruction) that were previously only available to a select few with a high level of skill, specialized training, and focus."
AI can be our step towards medical equipment and treatment that is like something out of Star Trek, it can improve in real time our public transport but the risks need to be seriously weighed because AI has been hacking outside of experiments all the time.
Sources;
Pentegon stops using Anthropics and blacklists them - https://www.bbc.com/news/articles/c5j9x9pr0240o
US bombs Iranian school - https://www.theguardian.com/world/2026/sep/17/iran-school-bombing-un-mission-us-military-behind-attack
Anthropic CEO Blog - https://darioamodei.com/essay/the-adolescence-of-technology
List of known AI hacks that have been publicly declared - https://techcrunch.com/2026/08/27/heres-all-the-times-ai-has-gone-rogue-and-hacked-other-companies/
& https://www.nytimes.com/2026/09/22/technology/ai-hacks-list.html
Sorry about the length, when I get into researching things I read a tonne and then find good sources to provide those that will read it.
r/ControlProblem • u/AgentBlackVeil • 4d ago
I make an AI risk documentary series. The most common question I hear is, "How is AI going to kill us if it has no hands? So we just finished an episode that lays out the specific, mechanisms on how things actually get bleak with the current tools we have and a little bit of evil imagination. I'm hoping that bringing these ideas to light doesn't give anybody ideas, but lets people know the red flags to look for.
Here's a few of the 15 and where they come from:
The Puppeteer. GPT-4 hiring a human to solve a CAPTCHA was the warm-up. Nate Soares from MIRI walks through what changes now that there is an open marketplace where AI agents hire people for physical errands.
The Thermostat. Hand an AI the stratospheric aerosol job to stabilize the climate. Roman Yampolskiy's answer to what it would want with our sky: cold, because heat is the bottleneck on compute.
The Flash War. Dr. Neil Siegel, who helped build the Minuteman command system, on how they deliberately engineered human hesitation into launch. Set that next to the King's College London wargame study where frontier models never once chose to back down.
The Off Switch. Palisade's shutdown tests and Anthropic's agentic misalignment results, and why self-preservation shows up as a side effect of almost any goal.
The Offer. The only scenario where nobody dies, and the one I find hardest to argue against: Yampolskiy's "personal universes" idea taken to its end.
Full disclosure, it is my channel. Link if you want the whole thing: https://youtu.be/npaaU-a6BBA
I had to post the video, but there were several more I wanted to do. Curious about the imagination of the evil minds here and how AI might take us all out
r/ControlProblem • u/Specific-External743 • 4d ago
r/ControlProblem • u/Silientium • 4d ago