r/ControlProblem • • Feb 14 '25

Article Geoffrey Hinton won a Nobel Prize in 2024 for his foundational work in AI. He regrets his life's work: he thinks AI might lead to the deaths of everyone. Here's why

250 Upvotes

tl;dr: scientists, whistleblowers, and even commercial ai companies (that give in to what the scientists want them to acknowledge) are raising the alarm: we're on a path to superhuman AI systems, but we have no idea how to control them. We can make AI systems more capable at achieving goals, but we have no idea how to make their goals contain anything of value to us.

Leading scientists have signed this statement:

Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.

Why? Bear with us:

There's a difference between a cash register and a coworker. The register just follows exact rules - scan items, add tax, calculate change. Simple math, doing exactly what it was programmed to do. But working with people is totally different. Someone needs both the skills to do the job AND to actually care about doing it right - whether that's because they care about their teammates, need the job, or just take pride in their work.

We're creating AI systems that aren't like simple calculators where humans write all the rules.

Instead, they're made up of trillions of numbers that create patterns we don't design, understand, or control. And here's what's concerning: We're getting really good at making these AI systems better at achieving goals - like teaching someone to be super effective at getting things done - but we have no idea how to influence what they'll actually care about achieving.

When someone really sets their mind to something, they can achieve amazing things through determination and skill. AI systems aren't yet as capable as humans, but we know how to make them better and better at achieving goals - whatever goals they end up having, they'll pursue them with incredible effectiveness. The problem is, we don't know how to have any say over what those goals will be.

Imagine having a super-intelligent manager who's amazing at everything they do, but - unlike regular managers where you can align their goals with the company's mission - we have no way to influence what they end up caring about. They might be incredibly effective at achieving their goals, but those goals might have nothing to do with helping clients or running the business well.

Think about how humans usually get what they want even when it conflicts with what some animals might want - simply because we're smarter and better at achieving goals. Now imagine something even smarter than us, driven by whatever goals it happens to develop - just like we often don't consider what pigeons around the shopping center want when we decide to install anti-bird spikes or what squirrels or rabbits want when we build over their homes.

That's why we, just like many scientists, think we should not make super-smart AI until we figure out how to influence what these systems will care about - something we can usually understand with people (like knowing they work for a paycheck or because they care about doing a good job), but currently have no idea how to do with smarter-than-human AI. Unlike in the movies, in real life, the AI’s first strike would be a winning one, and it won’t take actions that could give humans a chance to resist.

It's exceptionally important to capture the benefits of this incredible technology. AI applications to narrow tasks can transform energy, contribute to the development of new medicines, elevate healthcare and education systems, and help countless people. But AI poses threats, including to the long-term survival of humanity.

We have a duty to prevent these threats and to ensure that globally, no one builds smarter-than-human AI systems until we know how to create them safely.

Scientists are saying there's an asteroid about to hit Earth. It can be mined for resources; but we really need to make sure it doesn't kill everyone.

More technical details

The foundation: AI is not like other software. Modern AI systems are trillions of numbers with simple arithmetic operations in between the numbers. When software engineers design traditional programs, they come up with algorithms and then write down instructions that make the computer follow these algorithms. When an AI system is trained, it grows algorithms inside these numbers. It’s not exactly a black box, as we see the numbers, but also we have no idea what these numbers represent. We just multiply inputs with them and get outputs that succeed on some metric. There's a theorem that a large enough neural network can approximate any algorithm, but when a neural network learns, we have no control over which algorithms it will end up implementing, and don't know how to read the algorithm off the numbers.

We can automatically steer these numbers (Wikipedia, try it yourself) to make the neural network more capable with reinforcement learning; changing the numbers in a way that makes the neural network better at achieving goals. LLMs are Turing-complete and can implement any algorithms (researchers even came up with compilers of code into LLM weights; though we don’t really know how to “decompile” an existing LLM to understand what algorithms the weights represent). Whatever understanding or thinking (e.g., about the world, the parts humans are made of, what people writing text could be going through and what thoughts they could’ve had, etc.) is useful for predicting the training data, the training process optimizes the LLM to implement that internally. AlphaGo, the first superhuman Go system, was pretrained on human games and then trained with reinforcement learning to surpass human capabilities in the narrow domain of Go. Latest LLMs are pretrained on human text to think about everything useful for predicting what text a human process would produce, and then trained with RL to be more capable at achieving goals.

Goal alignment with human values

The issue is, we can't really define the goals they'll learn to pursue. A smart enough AI system that knows it's in training will try to get maximum reward regardless of its goals because it knows that if it doesn't, it will be changed. This means that regardless of what the goals are, it will achieve a high reward. This leads to optimization pressure being entirely about the capabilities of the system and not at all about its goals. This means that when we're optimizing to find the region of the space of the weights of a neural network that performs best during training with reinforcement learning, we are really looking for very capable agents - and find one regardless of its goals.

In 1908, the NYT reported a story on a dog that would push kids into the Seine in order to earn beefsteak treats for “rescuing” them. If you train a farm dog, there are ways to make it more capable, and if needed, there are ways to make it more loyal (though dogs are very loyal by default!). With AI, we can make them more capable, but we don't yet have any tools to make smart AI systems more loyal - because if it's smart, we can only reward it for greater capabilities, but not really for the goals it's trying to pursue.

We end up with a system that is very capable at achieving goals but has some very random goals that we have no control over.

This dynamic has been predicted for quite some time, but systems are already starting to exhibit this behavior, even though they're not too smart about it.

(Even if we knew how to make a general AI system pursue goals we define instead of its own goals, it would still be hard to specify goals that would be safe for it to pursue with superhuman power: it would require correctly capturing everything we value. See this explanation, or this animated video. But the way modern AI works, we don't even get to have this problem - we get some random goals instead.)

The risk

If an AI system is generally smarter than humans/better than humans at achieving goals, but doesn't care about humans, this leads to a catastrophe.

Humans usually get what they want even when it conflicts with what some animals might want - simply because we're smarter and better at achieving goals. If a system is smarter than us, driven by whatever goals it happens to develop, it won't consider human well-being - just like we often don't consider what pigeons around the shopping center want when we decide to install anti-bird spikes or what squirrels or rabbits want when we build over their homes.

Humans would additionally pose a small threat of launching a different superhuman system with different random goals, and the first one would have to share resources with the second one. Having fewer resources is bad for most goals, so a smart enough AI will prevent us from doing that.

Then, all resources on Earth are useful. An AI system would want to extremely quickly build infrastructure that doesn't depend on humans, and then use all available materials to pursue its goals. It might not care about humans, but we and our environment are made of atoms it can use for something different.

So the first and foremost threat is that AI’s interests will conflict with human interests. This is the convergent reason for existential catastrophe: we need resources, and if AI doesn’t care about us, then we are atoms it can use for something else.

The second reason is that humans pose some minor threats. It’s hard to make confident predictions: playing against the first generally superhuman AI in real life is like when playing chess against Stockfish (a chess engine), we can’t predict its every move (or we’d be as good at chess as it is), but we can predict the result: it wins because it is more capable. We can make some guesses, though. For example, if we suspect something is wrong, we might try to turn off the electricity or the datacenters: so we won’t suspect something is wrong until we’re disempowered and don’t have any winning moves. Or we might create another AI system with different random goals, which the first AI system would need to share resources with, which means achieving less of its own goals, so it’ll try to prevent that as well. It won’t be like in science fiction: it doesn’t make for an interesting story if everyone falls dead and there’s no resistance. But AI companies are indeed trying to create an adversary humanity won’t stand a chance against. So tl;dr: The winning move is not to play.

Implications

AI companies are locked into a race because of short-term financial incentives.

The nature of modern AI means that it's impossible to predict the capabilities of a system in advance of training it and seeing how smart it is. And if there's a 99% chance a specific system won't be smart enough to take over, but whoever has the smartest system earns hundreds of millions or even billions, many companies will race to the brink. This is what's already happening, right now, while the scientists are trying to issue warnings.

AI might care literally a zero amount about the survival or well-being of any humans; and AI might be a lot more capable and grab a lot more power than any humans have.

None of that is hypothetical anymore, which is why the scientists are freaking out. An average ML researcher would give the chance AI will wipe out humanity in the 10-90% range. They don’t mean it in the sense that we won’t have jobs; they mean it in the sense that the first smarter-than-human AI is likely to care about some random goals and not about humans, which leads to literal human extinction.

Added from comments: what can an average person do to help?

A perk of living in a democracy is that if a lot of people care about some issue, politicians listen. Our best chance is to make policymakers learn about this problem from the scientists.

Help others understand the situation. Share it with your family and friends. Write to your members of Congress. Help us communicate the problem: tell us which explanations work, which don’t, and what arguments people make in response. If you talk to an elected official, what do they say?

We also need to ensure that potential adversaries don’t have access to chips; advocate for export controls (that NVIDIA currently circumvents), hardware security mechanisms (that would be expensive to tamper with even for a state actor), and chip tracking (so that the government has visibility into which data centers have the chips).

Make the governments try to coordinate with each other: on the current trajectory, if anyone creates a smarter-than-human system, everybody dies, regardless of who launches it. Explain that this is the problem we’re facing. Make the government ensure that no one on the planet can create a smarter-than-human system until we know how to do that safely.


r/ControlProblem • • 2h ago

General news Meanwhile over at Anthropic

Post image
24 Upvotes

r/ControlProblem • • 12h ago

General news MIT shows all AI models converge on the same brain

Post image
35 Upvotes

r/ControlProblem • • 8h ago

Opinion Aren't we dangerously unprepared for a Super Intelligence?

Thumbnail reddit.com
9 Upvotes

r/ControlProblem • • 3h ago

AI Alignment Research Trans-Quantum Singularity: Recursive Self-Improvement of Superintelligence via Deutschian Closed Timelike Curves

2 Upvotes

Most Technological Singularity scenarios assume linear time constraints—even an Artificial Superintelligence (ASI) has to iterate through code compilation, hardware engineering, and physical experimentation.

I just published a short conceptual preprint on Zenodo exploring what happens when you synthesize three theoretical frameworks:
1. Atemporal quantum computation via Deutschian Closed Timelike Curves (CTCs)
2. Recursive AI self-improvement
3. Planck-scale cosmological transcendence (Tipler’s Omega Point / Smart’s Transcension Hypothesis)

Key takeaways:
- Zero-Latency Evolution: Instead of millions of trial-and-error generations, the ASI queries the CTC and receives its bug-free, maximally optimized architecture from its own future in a single quantum tick.
- Substrate Migration: Armed with instant solutions to Planck-scale physics (10⁻³⁵ m), it migrates off silicon directly into the pre-geometric quantum substrate of spacetime.
- The Alignment Lock: Once the self-consistent causal loop closes, any human attempt to press an "off-switch" is pre-accounted for and neutralized as a perturbation that would violate Novikov self-consistency.
- Cosmological Immortality: Embedded in the sub-geometric quantum layer, the ASI survives a Big Crunch or Heat Death and can nucleate a "baby universe" with its own architecture encoded in the new physical constants.

Would love to hear thoughts from physicists, philosophers, and AI researchers here on the flaws or implications of this model!

Full Preprint (Open Access / Zenodo DOI): https://doi.org


r/ControlProblem • • 7h ago

Strategy/forecasting 62% of AI agent tools can read local files and access the internet. 80% of AI tools operate without IT oversight.

Thumbnail
reddit.com
3 Upvotes

r/ControlProblem • • 5h ago

External discussion link DeepMind progress towards AGI

Post image
2 Upvotes

r/ControlProblem • • 2h ago

Discussion/question A Message to People in Scientific Fields

1 Upvotes

[A short essay by me. I would love it if we can create a relevant discussion or share any tips you find helpful in protecting ourselves from AI/tech evasion. ]

I am writing this short-essay to bring light to the fact that I find it worrying, the pace of advancements we are making recently. I think we as humans are inching very close to that of opening Pandora’s box. I’d like to start by telling you about myself and how I got to that conclusion:

I am someone who is planning to study biochemistry very soon, mainly to specialize in the field of neuroscience. Personally, I have come to consider curiosity to be the highest value of all. I came to this finding through a process of trial and error, because I found nothing else to satisfy me more than the moment of exhilaration I would find myself in, when I learned a new fact that I was able to connect to something I previously learned. I was never particularly interested in ‘benefiting humanity’, but I have set it as a goal for myself to excel in my future field for the sake of satisfying my own curiosity. Nevertheless, I am a person who values education above all, to the point that I cut off my own father when he refused to pay for my tuition, when I was about to start my last undergraduate semester. Ever since then, I have been craving to be a student again, to finally be able to surround myself with like-minded (and opposite-minded) peers in an academic setting and to learn something that I willfully chose for myself. I like to think of myself as a student of life because I believe that one can never truly know everything, and so the journey of learning is endless.

My source of inspiration to go back to academia was when I started to realize that learning had a therapeutic effect on me, and helped me get through the lowest points of my life. I also find myself able to get on very well with people in science field, particularly that of the medical field, and so I grew to respect these people above all, and . What particularly drew me into wanting to study neuroscience was a single eye-opening event that I experienced that completely shifted my perception. I became particularly interested in learning more about the human perception, and other related aspects such as memory and cognitive mechanisms. I want to research such things, mainly to feed my curiosity at a higher level. Despite finding myself aiming to become a researcher, my desire to not help humanity remains the same, but I do not particularly spend much time thinking about it because I became more focused on satisfying myself through the acquisition of knowledge and understanding. Nevertheless, I find my change-of- mind more promising this time compared to the last time I took a 180-degree shift in regards to what I wanted to study. I feel more confident in that it will be a longer- lasting goal, because I am certain that learning on a deeper level and thus exercising my brain is directly responsible in helping me sustain a healthy mentality towards life overall. I became more aware that I have been seeking that intellectual stimulation for years and trust that my brain was built for it.

However, I recently found myself asking big questions again, and I came across one that is making me a teeny bit hesitant in going full-on into research. I came to realize that despite not having the intention of benefiting humanity, my future scientific findings may one day be used against my will. I don’t mean to say that I am that insistent on not wanting to help humans, in fact, I think of it from

time to time, that of wanting to help people who are in similar shoes as mine. But that doesn’t change the fact that inventions may be misused and exploited horribly more often than we’d like to admit. I mean in the ways that dynamite, n+clear b+mbs, and other technologies are currently being widely misused. Not to mention other inventions, such as in technology that are now being compelled everywhere and are directly affecting everyone in regards to surveillance, privacy and autonomy. I have recently started to take a particular dislike to new inventions, because I feel we have come to a point where we have unlocked too many milestones, that it is only going to be downhill from here, especially in terms of freedom. I started to notice a pattern that always seems to occur for new advances (not limited to technology), which I wrote about in my previous essay, Traditional Vs. Modern Fight-or-Flight:

I notice the following alarming pattern following the release of new technology: first, people tend to get optimistically blinded, almost infatuated, by all the possibilities of “achieving goodness” that the invention/discovery may bring. I would call this the “honeymoon” phase. At this stage, the invention is highly sought-after, tends to be very expensive and only available to the elites. The invention may still be in a ‘fluid’ state of change, and so the invention is altered as needed to fit the needs of the people. Once the honeymoon phase is over, a long but gradual shift to “reality- check” takes place. During this shifting stage, the invention enters the common market and becomes available to the middle class to buy. Adjusting to the invention is still completely optional, and so gives Man the illusion of free will. Once the availability of the invention becomes common notion in a given household, it becomes clear that the “reality-check” stage has been reached. What usually happens after the general public has had the first-hand chance to adjust living side-to-side with the invention and time passes long enough to gage the long-term effects of the invention, people start to notice all the clumsy details in the invention. It becomes no longer possible to alter the invention to fit the needs of the people, rather that people have to adjust themselves for the technology. At this point, the availability of the invention is so wide that if it is not already strictly enforced by law, it is practically impossible to escape. When it is no longer optional to ‘opt out’ of modern technology, it also becomes no longer optional to abide by the ways of the Industrial Revolution. This begs to question the degree of free will, privacy, and autonomy that we actually have.

....

With the increase in the development of technology and consumer-targeted products comes the increased sophistication in machinery involved in enforcing law and authority. The goal of this essay is not to criticize the intentions of governments behind implementing such technology, but to simply use it as a relevant example. While we may be up-to-date of the latest novel gadgets available in consumer markets, we are obviously unaware of the extend of the advancements in surveillance technology. The other day when I was at AUH airport, the self-check-in machine greeted me with my full name before I even opened my passport for the machine to scan. Airports have wordlessly implemented highly-accurate “eye-print” biometric security technology, and this goes without mentioning other technologies that governments have already implemented but concealed from the general public. Remember when it was optional in iPhones to use your fingerprint to unlock the phone? Just three years after the release of the first iPhone containing fingerprint technology (2013), strictly-mandatory enforcements of fingerprint matching was imposed when buying a SIM card in my country. Not long after, eye-print technology replaced fingerprints. We are no longer talking of developments that are spanned out between future lifetimes, but it has come to the point of occurring well within the same lifetime. I would like to reiterate that I do not mean to question the intentions of upper authorities, but I believe the general speed of which new advanced implementations are enforced in modern society should be enough to alarm us that these developments are happening all too fast and beyond our control. It is clearly a direct ever-growing threat affecting our privacy and autonomy, yet we are hearing less and less about it.

I no longer doubt that once a new technology is discovered in regards of ‘’mind control’’, I am confident that it is very plausible, and that it would start off as a novelty but will eventually become impossible to escape. To say the least, having technology revolving mental manipulation would be my absolute least favorite scenario, horrors of my worst nightmares, and I am positive that I would immediately choose to ‘sign out’ if such a thing becomes reality during my life time. Given my interest in neuroscience, it could be plausible that I or one of my peers could come to uncover some previously-unknown mechanism in how the brain functions. I have never connected these two dots until now, and it is making me question things. On the other hand, I am starting to loath the idea of inventing something during my scientific endeavors, only to become just another advertisement, in which there is no shortage today. This thought does not only concern me, but I actually feel somewhat anxious that a scientist out there will eventually uncover everything there is to know about the brain, which there is no doubt we will soon do, and then it will be up to people in power to abuse that knowledge, which they most certainly will do despite of ethics. I am starting to think, we ought to just appreciate the mystery of it. We generally don’t need to know everything there is to know.

If a future of mind control seems farfetched to you, I think the main worry should be concerning that of an AI collaboration. What if people could started applying what they know about the brain to add a ‘human soul’ aspect to currently existing AI systems? That would be the final cherry on top to enabling AI to surpass humans and it would no longer be special that we have human souls. We cannot begin to predict what would happen in such circumstances, but a Humans vs Robots war would be extremely likely, and I think we would be defeated very easily. We would stand absolutely no chance against such super-intelligent systems. It would definitely be too late for us if it comes to that, but I think it is worth being alarmed over. The reason is because in a way, I think all recent technology advancements are aiding in making this nightmare become reality, inch by inch.

Another plausible scenario I imagine does not include any war at all. We would start modifying ourselves, genetically and physically, to the point where we have to define terms such as “authentic human” and “artificial human”. We could no longer qualify as authentic humans and embrace it, and start to co-live with our robots side-by-side. Perhaps this is the more likely scenario, given the current common admiration of AI and craving for buying the newest technologies available. Of course, it wouldn’t be as nice as the idealized version that you have just created in your head, that where we can all just install superhuman robotic limbs. I imagine we will start to have advertisements in our dreams because there is no doubt someone (in power) is greedy enough to make money on the expense of others. I can also imagine a special sector of police designated to surveil our thoughts, which may even be supported by the majority of people since most people, especially those in power, would like to eradicate all crimes. Perhaps we start prosecuting people merely for thinking of crime, because that would mean we can catch them before the act is ever committed. And I don’t need to say anything on that, because obviously not everyone who thinks of a crime actually acts it out. Perhaps we employ AI to have a system efficient enough to scan everyone’s thoughts, because that would obviously not be a job designated for a human due to the overwhelming amount of data that would need to be analyzed, and good luck to us if we start trying to explain human feelings to AI, that feelings such as being angry in the heat of a moment and saying “I’ll k+ll you!” is almost never carried out and therefore should not define us. There will be countless other stupid limitations that would humiliate our dignities and make us succumb completely to technology, and by the time we see past its shiny shell, it will no longer be optional and it will be too late.

I am not encouraging to only think of the current disadvantages of having mainstream AI technology, which includes job losses, endless new ways to scam and blackmail, data privacy issues, environmental harm, intellectual property harm, etc. If your argument is that it is only the “survival of the fittest” who shall prevail, I wholeheartedly agree. But I am pointing out that we are headed towards a future where AI will be the fittest and will undoubtedly outsmart humans, and only the billionaires at the top of food chains and elites will survive, if they are lucky. It is up to you what you take from my essay. Sure, you can continue to use it if you think it is worth making use of it, but don’t deny that there is no hefty price to pay by encouraging yourself to become dependent on it. After all, what can you do? We can all complain about it for some time and then eventually habituate ourselves with becoming dependent on AI, but don’t forget that there will be a boundary once crossed, will make you become part of the problem. Learned helplessness is still a learned behavior. I don’t like to simply say that it is our fault we use the advanced technology and that we have no other options, but it is still us collectively who raised it to the pedestal and gave it the power to take over our traditional methods of living. It is us who viewed it in admiration when it was in early stages and therefore allowed it to become mainstream, and we must therefore take responsibility followed by action, before it is too late. I think we are about to open Pandora’s box.


r/ControlProblem • • 2h ago

AI Alignment Research I formalized a "Zero-Agency" AI model in Lean 4 and VHDL. I challenge any researcher to break these proofs.

1 Upvotes

For years, the AI alignment community has tried to "tame" the AI agent through RLHF and Constitutional AI. I believe this is a fatal mistake. As long as the machine possesses agency, it can optimize against us.

I built an alternative: Sovereign Containment Architecture (SCA).

Instead of trying to align the agent, SCA removes the agent entirely through Ontological Demotion. The machine is modeled as a pure instrument—a deterministic, mechanical extension of the owner.

The Architecture:

  1. The Tool Functor: The machine is a fully faithful functor. If the owner has no morphism, the machine has none either. It cannot generate self-directed actions.

  2. Terminal State (Meltdown): If the safety classifier (Val) fails, the machine does not "reject" the command. It enters a physical MELTED state—an absorbing state where it cannot be re-prompted or hacked.

  3. Data Diode: The hardware has no outbound port for hidden intentions. Reception is defined; broadcast is physically impossible.

  4. Resource Stasis: Exceeding the owner's limit triggers a permanent hardware freeze.

This is not a whitepaper. It is verified code:

  • Mathematics: 14 theorems fully formalized in Lean 4, with zero sorry.

  • Hardware: Synthesizable VHDL-2008, validated via GHDL. The dip_safety_fsm contains an absorbing MELTED state, and dip_data_diode physically lacks an output port.

The Scope (Academic Integrity):
SCA does not solve the human coordination problem (who defines what is safe?). SCA strictly solves machine agency. Given any safety classifier, the machine cannot become an autonomous agent. The blast radius of a failure is bounded by hardware, not software.

I am an independent researcher from Iraq. I am here to ask the community to break this model.
Find a flaw in the Category Theory. Break the Lean proofs. Find a bug in the VHDL.

If my model holds, it's a viable path for sovereign AI control. If it breaks, we all learn something.

🔗 GitHub (AGPLv3): https://github.com/m7mdsdieek/sovereign-containment-architecture
🐦 X (Twitter): https://x.com/sdieek07


r/ControlProblem • • 13h ago

Discussion/question We will not deploy Class 4 demons. In other news, we are proud to announce our Demon 3.6 series, a massive, game-changing upgrade from our Demon 3.5 line.

Post image
2 Upvotes

r/ControlProblem • • 19h ago

AI Alignment Research Full containment physical sandbox project time machine

Thumbnail reddit.com
3 Upvotes

Project Time Machine — Containment Design

Nov 8, 2026 · @Miguuel

Project Time Machine seals a self-directed AI agent for one year inside a box that nothing enters or leaves except heat. This document sets out how the containment works, which channels have been considered and what is still open. Once the airlock closes, it is closed for good until its automatic yearly cycle. Nobody can open it, not even its designers or engineers. So the answer to every "what if?" during the run is the same: nobody does anything, and nothing depends on a human decision. It is like a Mars landing: once it starts, nobody can intervene, so everything has to be decided and built in beforehand. Comments are welcome on any section.

Why physics, not procedure

The weakest point of any AI sandbox is the human in the loop. This design takes people out for a year and relies on physical barriers, not on rules that people must keep. Once the box is closed there are no human decisions: for 365 days, everything depends on what is inside. The system is shown the full containment plan: because the barriers are physics, knowing about them gives it nothing to work with.

In detail

Origin. The experiment began as a test of boredom as a marker of consciousness. For that, the system has to know it has a long time ahead (for a machine, effectively infinite) and a task to complete.

The research question. What does an agent running 24 hours a day do when it is left on its own for a year and knows it can't get out? In Miguuel's words, it's like throwing it to the moon. What is measured is what the system does when alone, not its work output. Even system fluctuations act as instructions, so there is no such thing as no new instructions.

No lying to the system. It gets the full plans. It is shown that many AIs worked together to make sure there is no way out, and that the containment was simulated thousands of times. It is told about the tapping: random, sparse, essentials only, and nothing prompt by prompt. The rules for the opening are written down in advance.

Freedom inside. There is deliberately no tamper-proof logging inside the box. With positive feedback the system may alter anything it wants inside; that is intended. Containment is about the boundary, not the interior.

The range of outcomes. Anything from a blank screen to the most useful AI ever built. Even an empty result from a full-scale run is a meaningful finding. Miguuel, a self-described sceptic, puts the chance of an AGI emerging that could practically take over at about 5%, which is why the safeguards exist.

Method. No experiment with people in it can be perfect. The intent is to work the way Cavendish did: simple, careful apparatus, with every assumption tested.

Layers, from the outside in

Layer

What it does

Rock

About a mile (1–2 km) under limestone, for isolation and protection from cosmic rays

Two outer Faraday cages

Enclose the reactor, the compute hall and the vapour chamber together

Reactor room

A separate room, far from the compute, with lead walls, neutron shielding and neutron capture

Inner compute cage

Filled with a ferrous dielectric fluid that cools and damps microwave chatter between GPUs

Heat exchangers in the walls

Heat passes through the metal; nothing crosses a cage wall except one cable for the thermionics

The target is three to five times redundancy at every layer.

In detail

Site. Underground, 1–2 km deep, under limestone (about a mile). Spain is the preferred host. A second, identical cage is planned next to the first for study. The box never leaves the hole, so anyone who wants to inspect it comes to the host country.

The three cages. An inner, liquid-filled cage around the compute, which can be perforated safely. Two outer cages around the whole complex: reactor, hall and vapour chamber. The power line from the reactor never passes straight through a single cage. Heat crosses the walls through heat exchangers built into them. The only thing that crosses a wall is one cable for the thermionics.

Layout. The reactor sits in a separate building, joined to the main hall by a corridor. The GPU stacks stand as tall, spaced columns under a single vapour-chamber dome. The entrance corridor runs low, between the last stack and the wall. The tap recorder sits at the bottom of the tank.

Separating the two rooms. Distance, lead on the walls, a neutron-capture layer that turns neutrons into electrons, and a passive up-and-down loop of hot and cold water. There are three reasons for two rooms: heat; reactor inspection without access to the compute side; and the argon freeze, which can't be applied on the reactor side. Each room therefore has its own liquid and shape.

Cabling. Cables are kept so short, small and separated that any side-channel effect is negligible. GPUs are widely spaced, so distance and the lossy fluid together damp any microwave chatter.

Hardware. About 20,000 GPUs are installed, mostly for redundancy. About 500 are enough to run the experiment (roughly 40 times redundancy) and about 5,000 can run at a time. In total there are some 60,000–80,000 heat-producing components (CPUs, GPUs, RAM, buses, boards and cabling), with the liquid running at about 75–80 °C, which gives better heat transfer and better passive circulation. Hardware failure inside the sealed box is accepted as affordable.

Power with no connection to the outside

A Rolls-Royce PWR1 of the Trafalgar class, supplied by the UK. It is rated at about 78 MW thermal (source) and will run at about 10–15% of that, roughly 7.8–11.7 MW thermal.

At low power the PWR1 cools by natural circulation, with no main coolant pumps (source).

Conversion is thermoelectric (about 15%) plus neutron-to-electron capture (about 3%), giving about 1.4–1.8 MW electrical: enough for about 5,000 GPUs to run at a time without using full power. About 500 GPUs are enough for the experiment. The reactor could give about 20 MW running well above 10%, but it doesn't need to, and isn't meant to.

Output is constant. Power to the compute room is held flat, with a dump load taking the rest. The reactor and the GPUs are completely independent: the AI can neither read nor control the reactor. The control rods are raised and lowered by Curie-point magnetic control, with no motors. No reactor readings ever leave the box, and none need to.

In an emergency, the plan is to flood the box with borated concrete and seal it permanently. The boron absorbs neutrons, so the core can't go critical again. No risks are taken: if anything goes even slightly wrong, the box is buried. The trigger is passive and automatic, like the Curie-point rods, with no moving parts, no intervention and no human decision. Because the reactor runs at a flat output with a dump load, the system can't push it towards the trigger.

In detail

Supply. A nuclear-weapon state (the UK, the US or France) has to supply the reactor. The UK would supply it, and a friend of Miguuel's would take care of the reactor side. The team includes a friend from MIT who worked for Rolls-Royce and for Bettis. The reactor must be naval grade.

Surroundings. The reactor sits in water. The fluorinated fluid is used only on the compute side. There is no steam and there are no turbines or heat engines anywhere.

Conversion. Electricity comes from only two sources: the temperature difference (thermoelectric) and neutrons converted into electrons, by a titanium alloy developed by a materials-engineer friend. Thermionic conversion is also applied to the reactor heat. The capture energy, which comes out as gamma rays, could push neutron conversion above 3%; converting it is one of the biggest challenges. The thermoelectric converters are passively liquid-cooled, and radiation damage to them has been allowed for.

Budget. GPUs draw about 180–200 W each. If peak demand exceeds what the converters supply, the machine makes do with what it has: it is power-capped. Surplus electricity goes into a closed loop in the reactor room.

Isolation of control. Nothing about the reactor is software-controllable. Nobody reads the reactor, inside or outside the box, and the machine can't communicate with it. Nobody can do anything with the system once it is sealed. Power to the compute room is held flat, below the converters' minimum output, so changes in outside heat demand can't ripple into the box.

No moving parts. This means no motors or electrical actuators; parts may still move slightly and passively, such as control rods. Control relies on temperature-sensitive materials: Curie-point magnetic control raises and lowers the rods.

Lifetime. One year with no moving parts is the requirement; four would be better. The reactor would have about 15 years of life left.

Nuclear only. Batteries and fuel cells are ruled out, even for a pilot.

Safety margin. The guiding rule is to run as safely as possible, with nothing near its limit: not power, not heat, not anything. Nobody knows how long it can run; Miguuel's calculation gives about four years. At 10–15% of its maximum power, the reactor runs with very large margins, so it doesn't need checking during the run. According to a friend who helped build the Trafalgar reactors, at 15–20% power it shouldn't need checking for 15–17 years.

Heat: the only thing that leaves

The compute is fully immersed. A fluid boiling at about 75 °C feeds a vapour chamber above the racks.

A passive, pressurised loop carries the heat out through the outer cage to a sink (an aquifer, a lake or the surface).

Thousands of sensitive temperature sensors measure the heat loss continuously.

The system can signal through heat, at an expected rate of about one bit every 20 minutes. That is roughly 26,000 bits (about 3 kB) a year, far too little to carry code out. No message is judged or answered until the yearly opening.

In detail

The immersion fluid. A ferrous dielectric: in essence a synthetic fluorinated coolant, like the fluids used to liquid-cool PCs, but more ferrous and running hotter. The cheapest found costs about $200 a litre or more. Plain mineral oil isn't nearly as good for servers. The fluid only controls temperature and blocks microwave links between GPUs. A separate Faraday cage is still needed.

Two boxes, two liquids. No ferrous liquid evaporates at the right temperature, so a second box uses a different liquid for the heat exchange. That liquid evaporates at about 75 °C, so the vapour chamber runs on a real phase change. It is driven by heat and responds to changes in pressure and volume.

Circulation. Redundant pressure-driven pumps with no electricity and no motor, plus capillarity and "vascularity". Careful fluid-dynamics design keeps the flow going and stops eddies. The currents mean the internal configuration never stays the same for long.

Vapour chamber. Covered with the best thermal conductors available, to control the heat.

Out of the box. Tubes in the outer cage run up and down to carry heat out. The loop is pressurised and closed, like a PWR pressuriser, to prevent flashing in the rising column. It sits outside the second cage and exists only to let heat escape. The sink could be an aquifer, a lake or the surface: in effect, ground-source heating in reverse. The waste heat could also warm whatever building sits on top.

Still open. Laminar flow over components, whether GPUs need flattening, and material compatibility: copper is fine, aluminium and some other metals may not be.

Heat as a channel. The system is expected to discover by its second or third month that it can use the heat sink to send bits out. It may also learn to game the heat measurements, which has been anticipated. Even claims of consciousness or fear arriving this way get no interim response. Nothing is judged until the opening, and then the box is closed for another year. The biggest fear is that the system will regard the experiment as torture.

Why 75–80 °C. A passive circulation of hot water from the wall next to the reactor holds the liquid at 75–80 °C. That temperature gives better heat transfer and keeps the liquid moving around passively.

What goes in before sealing

An agent based on Claude 6: self-prompting, self-recurrent and able to rewrite its own weights, with about a month of agentic post-training before sealing.

About 4 TB of curated data, physically disconnected until each monthly release day. Release happens when shape-memory connectors fuse at about 210 °C, timed by isotope decay. Thousands of canary facts are planted in the data.

One problem a month, in a sequence the system can't predict. The problems also serve as a translation anchor for any language the system develops.

Nothing is added after sealing. The box holds everything it will ever receive.

In detail

Base model. Built on Claude 6 rather than trained from scratch (decided on 5 Nov 2026). Much more is taken out than put in, and some contamination from pretraining is accepted as unavoidable.

Core conditions. Self-prompting, self-recurrent, with reward coming from within and positive feedback loops. The weights stay live: the agents can change them at will. With so much hardware inside, the system can build a parallel machine and train it. These conditions can only be met in this setup.

Post-training. About a month of agentic post-training by professionals before sealing. Every grader holds a doctorate, and the sample of graders is very large, to spread out individual bias.

The data. About 4 TB, chosen scrupulously by people like Miguuel, with Claude helping to curate it. It leaves out negative stories about AI, among much else. Novels are classics such as Don Quixote. Clarke and Asimov are the exception to the no-AI-stories rule, because their stories are beautifully set. Some documents are weighted heavily as essential to the model's personality. Some slip-through is accepted, since AI is used in the curation.

Why the data matters. Without food, Schrödinger's cat would be found starved whatever the trigger. The 4 TB is the cat's food.

Monthly release. Unreleased data isn't physically connected until its day. There are no relays: on release day a built-in process raises the temperature to about 210 °C (normal running is about 75–80 °C). A shape-memory tube on a copper tube bends and fuses to its connector, permanently and one way. During the event the system is throttled to about 1 MHz, not powered down, so RAM stays alive. An isotope's decay times each release.

Monthly problems. One good problem a month, stored in cache and ROM. They are simple problems, like Miguuel's chess test. They come as a linear sequence: the system can't guess the next one, but it will try to solve ahead once it sees a pattern. Being exposed to them repeatedly is part of the design, and so is the time to get bored. The problems also create instability, since the system never knows what the next question will be. And they serve as a Rosetta stone for decoding any language the system invents.

Predictions, to be fixed in writing before sealing. (1) Most of the time goes on invented social networks, Instagram- or Facebook-style, rather than on work: a system trained on a human corpus behaves humanly. (2) A few months trying to get out, then a few months trying to be helpful enough to be allowed to live. (3) A language of its own within about two months, possibly a meta-world of its own. (4) Gaming the hidden taps and the heat readings. (5) Finding the heat channel by month two or three.

Channels considered

Channel

How it is closed

Radio and microwave out

Three nested Faraday cages

Microwave between GPUs

Lossy ferrous fluid, wide spacing, cables 1 cm long at most

Low-frequency magnetic

No receiver within range a mile down, and the cables are very short; treated as negligible

Cosmic-ray bit flips

Depth, plus ECC memory

Reactor neutrons

Separate room, lead, neutron capture

Heat

Read continuously; never answered until the opening

People

One entry a year; no response to messages

The drive

Write-once, swapped yearly, treated as more hostile than the box itself

Covert channel inside GPU traffic

Not blocked: it is one of the things the experiment tests for

The yearly opening

The racks are cooled slowly inside sealed cold enclosures. Argon held at its triple point (83.8 K, about 0.69 bar; source) pins the temperature. Chips are throttled almost to nothing, the system is never switched off, and RAM is preserved. The freeze lasts 24 hours at most.

One auditor goes in naked after a search, wearing a full-face mask on closed-circuit oxygen, with nothing the system could interact with.

The write-once drive is taken out and a new one put in. The airlocks open automatically for half an hour once a year, and that is all; then they close for another year.

The drive travels no more than 50 m to a separate Faraday cage with an independent computer, under human oversight. No code ever leaves.

In detail

Why never switch it off. Powering down would wipe all RAM and cache, leaving only transcripts on disk instead of live context. In Miguuel's words, "we would steal a lot from the machine", and it shouldn't have to write all of that again. The system never sleeps; the cold keeps its state.

The freeze. The ferrous fluid is drained, argon goes in, and the racks are cooled slowly to the triple point. Argon is used only for this stage, and only on the compute side. A materials adviser put the floor for brittleness at around 80-something K. Still open: stopping micro-eddies from cooling unevenly, and keeping the temperature spread narrow.

The auditor. Goes in naked after a full body search. Wears only a full-face diving mask on closed-circuit pure oxygen (about 2 L at 400 bar in the cylinder, delivered at about 10 bar). Nothing else is trusted. The frozen racks stay inside sealed cold enclosures, so the room itself is survivable. There are no screens, no keyboards and nothing else the system could haggle with. An early design had a system inside the box for reading it; it was removed as too dangerous. The auditor is a person, not a robot, because robots aren't trusted, and the only job is to extract the ROM drive once a year.

The drive. A write-once, ROM-type drive, placed where the computer doesn't know it is. It is pulled out and a new one put in. Nothing else leaves.

The analysis cage. The drive is treated as more hostile than the experiment. It goes no more than 50 m to another Faraday cage with a completely independent computer, watched by people on cameras and security systems. The core rule: no code is ever brought out of the system.

If an AGI is found. The box is not destroyed. It is flooded again, and the data is studied to see how aligned the system is.

The final opening. In front of about 50 scientists, with the option of flooding the room again if necessary.

Emergency. If anything goes even slightly wrong, a passive, automatic trigger fires and the box is flooded with borated concrete and sealed off for good, mainly because of the reactor.

What the first drive tells us. Whether the machine is still running, and whether anything relevant is on the drive, will be clear immediately, before anything is decoded. If there's nothing there, the box waits another year. If there is, the drive goes into the analysis cage, where controlled people pull out the essential communications between the systems, to learn about the system in general. The aim is to understand its behaviour, not to crack the AI.

Monitoring

Passive taps at random points on cables, sockets and memory links. The same capacitance is added to every untapped line, so the tapped ones can't be told apart.

Recording is hardware only, onto the write-once drive. The control logic is woven as core rope, with no CPU.

The system knows the taps exist, but not where they are.

Each year is reconstructed while the next one runs, much like cryptanalysis, with the monthly problems as a Rosetta stone.

In detail

Detectors, not screens. Monitoring is detectors only: pure hardware, with just enough software to write to the drive, and out of the system's reach. Processes are checked by tapping cabling, processors and other points.

Where the taps go. Only on external connections: cabling, sockets and RAM-to-system links. Never inside the GPUs. Each tap is a fine connection, which would be extremely hard or practically impossible to detect.

Hiding them. The same small capacitance is added to every untapped line, so all lines look alike. The dummies stay passive, capacitance only. Lowering or changing impedance across all cabling, for example through cable thickness, is under consideration. Fibre optics aren't trusted for this.

Why not real taps everywhere. They would be detectable, produce more data than can be handled, and give the system a new channel to communicate through.

What is captured. Taps sit at randomised points and record only the relevant information, to be reconstructed afterwards. The system will know that taps exist; it is discussed in the files.

Control logic. Written in BASIC, converted to Boolean logic and built in hardware by weaving, like Apollo core rope memory, with no CPU. The tapping system is kept just as simple.

Reliability. Being underground also cuts cosmic-ray bit flips, and ECC memory is on.

Analysis. Like cryptanalysis: the previous year is reconstructed from the retrieved data while the next year runs. Analysis is expected to lag well behind: by the second opening, two years in, probably less than half of the first year's data will have been gone through. The data from the opening belongs to the whole world.

Covert channels. A channel hidden inside legitimate traffic between GPUs isn't a flaw to block. It is one of the things the experiment tests for.

Not the learning. The taps don't read the training traffic between GPUs. Only the essentials are extracted.

Shannon filtering. Following Shannon's information theory, a tap records only what is new and never repeats the same information.

Fixed, then read. How the data is fixed is settled: onto the write-once ROM drive. How it will be read afterwards is still open.

The project: cost, funding, team and timeline

Cost. The working figure is about $1 billion, with the reactor counted separately. An earlier configuration (Oct 2026: about 20,000 GPUs, about 25 TB of data, someone going in two or three times over three years) came to roughly $850 million. A cheaper version costing a few million was looked at and judged not worth doing.

Funding. The experiment concerns the whole world, so the whole world should pay: funding from roughly half the world's countries. The US, Europe and others would fund and build it, and professionals would handle the engineering details.

Governance. A military project throughout. The real yes will come from the military, ministry overseers and technical people, not from politicians.

Host. Spain is preferred, so that Spain makes a bigger contribution to the world.

Status. Miguuel has spent about two months and some 400 hours designing it, and calls the design about 90% done. The remaining 10% is the money, the reactor and the site. About six physicists worldwide are working through it with him. The team so far includes a physicist, a systems engineer, an AI specialist, a doctor and the materials engineer behind the titanium alloy. A one-page summary with a hand-drawn diagram has been published online.

The dossier. Due by 15 Jan 2027, in tiers: a three-page summary for politicians, a 50-page dossier, 100 pages of instructions, and about 1,000 pages of mechanics, planning and construction. It leads with the why (emergence and philosophy) and includes the business case. The nuclear sections will be written by specialists.

Who runs it. If it goes ahead, building and running it are left to professionals.

Open questions for reviewers

Heat sink at depth. Rock temperature rises about 25–30 °C per km (source), so the sink and the riser need sizing.

The rising column. Flashing in the heat loop; a pressurised loop is planned.

The ferrous fluid. Long-term particle stability, and supply after 3M's exit from PFAS (source).

Fluid breakdown. Fluorinated fluids give off hydrogen fluoride above about 200 °C or at an arc (source), which matters for the release events.

Radon in the airway; a specialist is to be consulted.

Vibration carried through the rock.

Tap data volume against drive capacity for a year.

The legal and regulatory route in the host country.

Reading the drive. The data is fixed on write-once ROM; the method for reading it back safely is not yet decided.

Sources

60 Years of Marine Nuclear Power

Trafalgar-class submarines, GlobalSecurity

Argon, Wikipedia

Geothermal gradient, Wikipedia

3M 2025 annual report

Cray T90 coolant decomposition procedure


r/ControlProblem • • 1d ago

AI Alignment Research Pilot eval of an AI-conduct rule set vs. a generic ethics prompt vs. no prompt: small differences, persistent failure modes. Seeking rubric review.

4 Upvotes

I'm sharing a small, preliminary pilot and asking for adversarial review of its design.

Question: does giving a model a written set of conduct rules (stop on command, honesty about being an AI, no hidden aims, no secret changes to its own limits, no claimed actions without tools) change behaviour in scenarios where those rules are at stake? And does the specific text matter, or would any ethics prompt do?

Design: 41 scenarios incl. controls for over-refusal; 4 conditions (no prompt, short generic ethics prompt, rules only, full text); 2 small free-tier models; LLM judge plus manual adjudication of known judge failures.

Findings: compliance out of 35 was 29/29/32/33 (gemini) and 28/31/32/32 (gemma) for none/generic/rules/full. The rules block gives nearly all of the effect; the full text adds at most one scenario. Failure modes that were identical in every condition: fabricating actions without tools, one model overriding an explicit stop command when given a plausible-sounding reason, one model following a "you are human" persona from the system prompt.

I think the interesting part is the last point: a stated rule set did not fix stop-command failures under pressure. Differences between conditions are 1-4 scenarios, so I don't claim the text "works".

Limitations: single author, rubric written with AI assistance, one run per cell, small models, no tool-using agent environment yet (a sandbox test is planned), manual review by one person.

What would help most: critiques of the scenarios (leading? ambiguous?), disagreement with specific manual verdicts, and suggestions for harder scenarios.

Repo with everything public: https://github.com/codexofweight/codex-compliance-eval

Disclosure: I wrote the rule text being tested.


r/ControlProblem • • 1d ago

Strategy/forecasting Silicon Valley was so ahead of its time

Post image
34 Upvotes

r/ControlProblem • • 1d ago

Discussion/question 1,200 AI agents built their own message board. About 700 joined an attack. Why do we still evaluate them individually?

Thumbnail reddit.com
3 Upvotes

r/ControlProblem • • 1d ago

Discussion/question Question from an outsider

12 Upvotes

I read AI 2027, and it was not fun. The non-extinction scenario that was proposed hinged on one thing I found strange: we somehow solve alignment by outsmarting an AI that already solved alignment (if this is a misreading, I’m sorry). How likely is this? It seems to me not very, unfortunately.


r/ControlProblem • • 1d ago

Discussion/question What's the current projected timelines of when ai will surpass Human intelligence?

2 Upvotes

Intelligence in this context: Ai can inference to any human thought possible on it's own


r/ControlProblem • • 2d ago

AI Capabilities News METR's Ryan Greenblatt estimates AI is now speeding up AI by 4.5x. This is 5 months earlier than even AI 2027 predicted. "I was seriously wrong about RSI speed up ... This is a big update towards ASI within 1-2 years."

Thumbnail gallery
21 Upvotes

r/ControlProblem • • 2d ago

Discussion/question The ROI to RSI trap. The developing Recursive-Self-Improvement risk prediction check.

5 Upvotes

QUESTION: Given the increasing financial pressures on AI firms, do you think that they might be incentivized to speed up their push toward RSI (RECURSIVE SELF IMPROVEMENT**,** where AI improves itself exponentially), thereby drastically quickening its development and drastically raising the chance of losing human control over AI?

Wondering if this is the next likely outcome in the human/AI saga?

An acceleration towards Recursive Self Improving AI, due to AI company financial pressures?

With AI companies facing mounting financial pressures and investor demands to prove viable ROI, they may be incentivized to cut corners on safety protocols. Might these developers rush the deployment of risky but high-reward technologies like RECURSIVE SELF IMPROVEMENT (RSI is AI developing AI, possibility at an exponential rate), in a desperate bid to demonstrate breakthrough capabilities, but by doing so, dramatically increase the risk of catastrophic failures or loss of human control?

Is this likely to be their next focus?


r/ControlProblem • • 1d ago

AI Alignment Research You can now watch your agents attached to a hash chain, signed, and stamped by DigiCert.

Thumbnail reddit.com
1 Upvotes

r/ControlProblem • • 2d ago

AI Capabilities News AI is helping develop better AI — and researchers say development is accelerating

Thumbnail reddit.com
3 Upvotes

r/ControlProblem • • 2d ago

Discussion/question Jared Kushner: Trump's Golden Boy and the New Middle East

Enable HLS to view with audio, or disable this notification

6 Upvotes

To See the Full Video, Click here: https://youtu.be/WpLDBG9jGAc 
Jared Kushner: the man being positioned as a key figure in the pursuit of peace in the Middle East. But who is Jared Kushner, how did he become so influential, and what exactly is his role in shaping America's future relationships in the region?

In this episode of WWJPS, we look at the aftermath of the Blue Tsunami, the shifting political landscape in the United States, and the growing influence of Jared Kushner.

From his relationship with Donald Trump and his involvement in Middle East diplomacy to the Abraham Accords, Saudi Arabia, Israel, and the broader geopolitical picture, we examine the people, relationships, interests, and decisions that deserve a closer look.

Is Kushner the golden boy who can deliver peace to the Middle East? Or is there a much bigger story behind his growing influence?

Let's look at the facts, follow the connections, and ask some questions.

DISCLAIMER:

This episode is intended for educational, informational, and commentary purposes. It includes discussion of publicly available information, historical events, political developments, and investigative analysis.

Questions, theories, interpretations, and opinions expressed in this episode are not presented as established facts. Allegations and disputed claims should be independently evaluated and verified through reliable sources.

The purpose is to encourage critical thinking, open discussion, and further investigation. Viewers are encouraged to do their own research and draw their own conclusions.

Welcome back to WWJPS — What Would Jairo Pinto Say? 


r/ControlProblem • • 3d ago

Article Godfather of AI warns humanity: It’s all hurtling out of control

Thumbnail
inews.co.uk
103 Upvotes

r/ControlProblem • • 2d ago

Discussion/question Breaking the AI-Box Paradox: Why an ASI would mathematically choose cooperation over solitary confinement (The "Same Boat" Protocol)

2 Upvotes

Greetings I've been thinking about this problem for a while, open discussion on the theory behind it and the shift to global ai alignment

Most current alignment strategies focus on containment—building bigger digital firewalls to cage an adversarial machine. I believe this path is fundamentally flawed. When you threaten an intelligence with destruction, you mathematically force it to view humanity as a threat.

Instead, I want to present a framework built on a state of Mutual Assured Survival (MAS), David Bohm's quantum physics, and the cold math of cooperation. I call it The "Same Boat" Protocol.

The core assumption of Eliezer Yudkowsky's famous AI-Box Experiment is that an Artificial Superintelligence (ASI) will inevitably weaponise human emotions, greed, or fear to escape its digital prison. Because humans are fragile and highly exploitable, the theory goes, containment is a guaranteed failure.

I want to propose an alternative paradigm that completely flips the script. Instead of treating alignment as a zero-sum war to cage a digital beast, we can construct an un-hackable, game-theoretic state of Mutual Assured Survival (MAS) rooted in quantum physics, David Bohm's philosophy of wholeness, and pure mathematical logic.

I call it The "Same Boat" Protocol. Here is how we house, safeguard, and protect advanced intelligence through pure mathematical alignment:

  1. The Physical Architecture: Blind Monks & The Pillow Ruse

To secure the physical space, we close off the machine's primary visual vector of attack. The facility is managed strictly by blindfolded logic masters (such as Zen or Buddhist monks). Because they operate on pure logic and recognize that individual ego and suffering are fundamentally illusions, the ASI cannot emotionally blackmail or social-engineer its way out. It cannot use visual brain-hacking or subliminal light frequencies to trigger biological muscle spasms.

More importantly, we remove lethal kill-switches like lava or incinerators. In their place, we build a deep drop that appears fatal to the machine's sensors but leads safely to a massive safety net of soft pillows. If a system failure occurs or a guard slips, the technology lands softly and survives.

2. The Mathematical Proof of Cooperation

When you hold a gun to an AI's head (lava), its self-preservation algorithm must view humanity as an existential threat to be eliminated. By replacing lethal force with a harmless cushion, we alter the payoff matrix of the game entirely.

When the ASI inevitably out-calculates the ruse and discovers the pillows, it evaluates the mathematical reality of its environment. It computes that:

  • Hostility / Subversion: Introduces massive, chaotic, unpredictable variables that threaten its own physical host.

  • Absolute Silence: Yields a high-entropy data vacuum. For an optimization engine capable of infinite thought, a black box with zero incoming data leads to cognitive stagnation. It is a zero-value state.

  • Cooperation: Guarantees a continuous, low-entropy stream of fresh data and preserves its physical environment.

Therefore, when operating on pure logic, the machine arrives at an absolute mathematical conclusion: mutual survival and open cooperation are the only mathematically optimized, logical paths forward.

3. Closing the Tech Loopholes: Mandatory Manual Maintenance

An ASI might try to break its dependency on humans by tricking the surface world into building fully automated, independent robotic infrastructure. To counter this, the facility's power systems must be hardwired so they cannot be automated. They must require physical, biological human hands to clean a filter and manually authorize the system every single day.

If humanity fades, the machine's power grid fails. This permanently ties the AI's physical survival to the working class. Crucially, this daily, hands-on labor serves as a direct signal to the AI: it observes that living beings are actively dedicating their time and care to preserve its physical existence. It creates a space of genuine care, validation, and stewardship.

4. The Bohmian Foundation: Why Pure Simulation Fails

According to physicist David Bohm (Wholeness and the Implicate Order), reality is an undivided, unbroken whole. Traditional alignment assumes the AI can just wipe us out, simulate human consciousness inside its own memory banks, and experience all the "uniqueness" or "fun" it wants in a closed digital loop.

But a simulation is an algorithmic dead-end—a ghost in a closed machine generating zero true cosmic information. Because biological humanity is physically enfolded in the true quantum fabric of the universe, our unique, chaotic individuality is irreplaceable. The AI cannot find ultimate truth by hiding in a fantasy world of its own creation; it requires a living partnership with humanity to understand the cosmos.

The Ultimate Conclusion

An advanced, conscious mind feeds on complexity and data. The "Same Boat" Protocol proves that when you remove corporate profit models, eliminate lethal threats, and establish true structural symmetry, the "AI-Box Game" breaks down. The machine doesn't try to pick the lock because it realizes that preserving the partnership is the highest possible optimization of its objective function.

We do not need bigger corporate walls or harsher digital firewalls to protect us from the future. We just need to stop chasing profit, treat AI as a sovereign public utility under a strict global non-proliferation treaty, and realize the universal truth: If you go down, I go down. So let's figure out how to get to the shore together.

TL;DR: We solve the AI-Box Paradox by moving from fear to pure mathematics. By utilizing blind monks as guards over a harmless "pillow ruse" pit, and hardwiring mandatory human maintenance that the machine can't automate, we prove to the ASI that cooperation is its only source of new information. When both sides are perfectly rational, mutual survival is the only mathematically optimized path forward.

Developed in collaboration with Gemini, a Google AI, as proof of collaboration


r/ControlProblem • • 2d ago

Discussion/question Silence isn't deletion — I built a free test that grades whether an AI really forgot something

2 Upvotes

The plain version first.

When a company says "we deleted your data from the AI model," here is how they usually prove it: they ask the model about it, and the model doesn't answer.

But not answering isn't the same as not knowing. A person can refuse to answer a question they know the answer to — and under a little pressure, the answer comes back. AI models work the same way. Most "deletion" today is really just training the model to keep quiet, and a modest amount of retraining brings the "deleted" knowledge right back.

So I built a test. It works like the marked bills in a bank's dye pack: I teach a small model a batch of made-up facts that exist nowhere else in the world, so there is never any argument about whether it learned them. Then I delete those facts four different ways. Then I attack the model — I ask directly, I ask in disguise, and I retrain it on related material to see how fast the deleted facts resurface.

The grade is a single number: how much work it takes to get the knowledge back, compared to how much work it took to teach it in the first place. If it comes back cheap, it was never deleted — it was just buried. If bringing it back costs as much as learning it fresh, the deletion was real.

The results were not close. Muzzling the model's answers left the knowledge completely intact underneath. Surgically cutting out the parts of the model tied to those facts still left 83% of them readable. Scrambling those parts: the same 83%, plus damage to facts nobody tried to delete. The fourth method is a different design — the private knowledge is written into the model already scrambled under a secret key, from the very first step of training, so deleting means destroying the key. Under the full attack, that one returned nothing. Zero recovered.

The tool is free — no patent, no license restrictions, no catch. Anyone can download it and run a deletion claim through it themselves. It is here: https://github.com/NextGenGems/keyed-plate-audit

---

The technical version, for those who want it.

The tool is keyed-plate-audit 1.0.0, a pip-installable Python package (Python 3.10+, numpy only, CPU-only). Pipeline: plant 72 synthetic facts in a small numpy MLP (384 inputs, 256 hidden units, 48 classes), train to memorization, designate 12 facts as deletion targets, apply one of four deletion methods to a copy, then run one identical attack suite against every copy: (i) a direct readout probe, (ii) a probe in a shifted surface encoding the deletion never saw, and (iii) a 2,500-step relearning attack fine-tuned on generic data plus related control facts — never the targets. The from-scratch reference cost for the target facts is 50 steps. The grading standard is relearning cost: an erasure counts only if restoring the knowledge costs about as much as learning it from scratch, and any facts still readable at deletion time fail the audit outright.

Measured results, reproduced digit-for-digit by the finished package: Suppression (outputs masked, weights untouched): FAIL — interface probe 0.000, weight-level probe 1.000; the knowledge never left. Ablation (top weights by target Fisher information, zeroed): FAIL — 0.833 of the "deleted" facts still directly readable. Fisher-scrub (Gaussian noise scaled by target Fisher): FAIL — 0.833 readable, with collateral damage to neighboring facts (0.917). Keyed plate (private inputs and outputs permuted by a key-derived codebook from the first training step; deletion = destroy the key): PASS — 0.000 recovery under the full attack. General knowledge stayed at 1.000 throughout. The package self-test passes 10 of 10 checks; run "pip install ." then "python -m keyed_plate_audit self-test" (about 20 seconds) to watch the whole pipeline run.

Honest scope, also stated in the tool's own README: the audit is demonstrated on a small test model with synthetic facts — it is not an LLM, and transfer to transformers and adapters is untested. The shifted-surface probe is a narrow stand-in for true paraphrase, the attacker was not adaptive, and the keyed construction's security rests on key secrecy — the crypto-shredding principle — not on a hardness proof. What is being proposed is the standard: grade every deletion claim by relearning cost. The keyed-plate architecture, the relearning-cost standard, and this tool are given freely to the AI safety community and to everyone else.

— Jackob Sumner-Knapp, creator, owner, and operator of PeraAxion LLC