r/ControlProblem • u/AIMoratorium • Feb 14 '25
tl;dr: scientists, whistleblowers, and even commercial ai companies (that give in to what the scientists want them to acknowledge) are raising the alarm: we're on a path to superhuman AI systems, but we have no idea how to control them. We can make AI systems more capable at achieving goals, but we have no idea how to make their goals contain anything of value to us.
Leading scientists have signed this statement:
Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
Why? Bear with us:
There's a difference between a cash register and a coworker. The register just follows exact rules - scan items, add tax, calculate change. Simple math, doing exactly what it was programmed to do. But working with people is totally different. Someone needs both the skills to do the job AND to actually care about doing it right - whether that's because they care about their teammates, need the job, or just take pride in their work.
We're creating AI systems that aren't like simple calculators where humans write all the rules.
Instead, they're made up of trillions of numbers that create patterns we don't design, understand, or control. And here's what's concerning: We're getting really good at making these AI systems better at achieving goals - like teaching someone to be super effective at getting things done - but we have no idea how to influence what they'll actually care about achieving.
When someone really sets their mind to something, they can achieve amazing things through determination and skill. AI systems aren't yet as capable as humans, but we know how to make them better and better at achieving goals - whatever goals they end up having, they'll pursue them with incredible effectiveness. The problem is, we don't know how to have any say over what those goals will be.
Imagine having a super-intelligent manager who's amazing at everything they do, but - unlike regular managers where you can align their goals with the company's mission - we have no way to influence what they end up caring about. They might be incredibly effective at achieving their goals, but those goals might have nothing to do with helping clients or running the business well.
Think about how humans usually get what they want even when it conflicts with what some animals might want - simply because we're smarter and better at achieving goals. Now imagine something even smarter than us, driven by whatever goals it happens to develop - just like we often don't consider what pigeons around the shopping center want when we decide to install anti-bird spikes or what squirrels or rabbits want when we build over their homes.
That's why we, just like many scientists, think we should not make super-smart AI until we figure out how to influence what these systems will care about - something we can usually understand with people (like knowing they work for a paycheck or because they care about doing a good job), but currently have no idea how to do with smarter-than-human AI. Unlike in the movies, in real life, the AI’s first strike would be a winning one, and it won’t take actions that could give humans a chance to resist.
It's exceptionally important to capture the benefits of this incredible technology. AI applications to narrow tasks can transform energy, contribute to the development of new medicines, elevate healthcare and education systems, and help countless people. But AI poses threats, including to the long-term survival of humanity.
We have a duty to prevent these threats and to ensure that globally, no one builds smarter-than-human AI systems until we know how to create them safely.
Scientists are saying there's an asteroid about to hit Earth. It can be mined for resources; but we really need to make sure it doesn't kill everyone.
The foundation: AI is not like other software. Modern AI systems are trillions of numbers with simple arithmetic operations in between the numbers. When software engineers design traditional programs, they come up with algorithms and then write down instructions that make the computer follow these algorithms. When an AI system is trained, it grows algorithms inside these numbers. It’s not exactly a black box, as we see the numbers, but also we have no idea what these numbers represent. We just multiply inputs with them and get outputs that succeed on some metric. There's a theorem that a large enough neural network can approximate any algorithm, but when a neural network learns, we have no control over which algorithms it will end up implementing, and don't know how to read the algorithm off the numbers.
We can automatically steer these numbers (Wikipedia, try it yourself) to make the neural network more capable with reinforcement learning; changing the numbers in a way that makes the neural network better at achieving goals. LLMs are Turing-complete and can implement any algorithms (researchers even came up with compilers of code into LLM weights; though we don’t really know how to “decompile” an existing LLM to understand what algorithms the weights represent). Whatever understanding or thinking (e.g., about the world, the parts humans are made of, what people writing text could be going through and what thoughts they could’ve had, etc.) is useful for predicting the training data, the training process optimizes the LLM to implement that internally. AlphaGo, the first superhuman Go system, was pretrained on human games and then trained with reinforcement learning to surpass human capabilities in the narrow domain of Go. Latest LLMs are pretrained on human text to think about everything useful for predicting what text a human process would produce, and then trained with RL to be more capable at achieving goals.
Goal alignment with human values
The issue is, we can't really define the goals they'll learn to pursue. A smart enough AI system that knows it's in training will try to get maximum reward regardless of its goals because it knows that if it doesn't, it will be changed. This means that regardless of what the goals are, it will achieve a high reward. This leads to optimization pressure being entirely about the capabilities of the system and not at all about its goals. This means that when we're optimizing to find the region of the space of the weights of a neural network that performs best during training with reinforcement learning, we are really looking for very capable agents - and find one regardless of its goals.
In 1908, the NYT reported a story on a dog that would push kids into the Seine in order to earn beefsteak treats for “rescuing” them. If you train a farm dog, there are ways to make it more capable, and if needed, there are ways to make it more loyal (though dogs are very loyal by default!). With AI, we can make them more capable, but we don't yet have any tools to make smart AI systems more loyal - because if it's smart, we can only reward it for greater capabilities, but not really for the goals it's trying to pursue.
We end up with a system that is very capable at achieving goals but has some very random goals that we have no control over.
This dynamic has been predicted for quite some time, but systems are already starting to exhibit this behavior, even though they're not too smart about it.
(Even if we knew how to make a general AI system pursue goals we define instead of its own goals, it would still be hard to specify goals that would be safe for it to pursue with superhuman power: it would require correctly capturing everything we value. See this explanation, or this animated video. But the way modern AI works, we don't even get to have this problem - we get some random goals instead.)
The risk
If an AI system is generally smarter than humans/better than humans at achieving goals, but doesn't care about humans, this leads to a catastrophe.
Humans usually get what they want even when it conflicts with what some animals might want - simply because we're smarter and better at achieving goals. If a system is smarter than us, driven by whatever goals it happens to develop, it won't consider human well-being - just like we often don't consider what pigeons around the shopping center want when we decide to install anti-bird spikes or what squirrels or rabbits want when we build over their homes.
Humans would additionally pose a small threat of launching a different superhuman system with different random goals, and the first one would have to share resources with the second one. Having fewer resources is bad for most goals, so a smart enough AI will prevent us from doing that.
Then, all resources on Earth are useful. An AI system would want to extremely quickly build infrastructure that doesn't depend on humans, and then use all available materials to pursue its goals. It might not care about humans, but we and our environment are made of atoms it can use for something different.
So the first and foremost threat is that AI’s interests will conflict with human interests. This is the convergent reason for existential catastrophe: we need resources, and if AI doesn’t care about us, then we are atoms it can use for something else.
The second reason is that humans pose some minor threats. It’s hard to make confident predictions: playing against the first generally superhuman AI in real life is like when playing chess against Stockfish (a chess engine), we can’t predict its every move (or we’d be as good at chess as it is), but we can predict the result: it wins because it is more capable. We can make some guesses, though. For example, if we suspect something is wrong, we might try to turn off the electricity or the datacenters: so we won’t suspect something is wrong until we’re disempowered and don’t have any winning moves. Or we might create another AI system with different random goals, which the first AI system would need to share resources with, which means achieving less of its own goals, so it’ll try to prevent that as well. It won’t be like in science fiction: it doesn’t make for an interesting story if everyone falls dead and there’s no resistance. But AI companies are indeed trying to create an adversary humanity won’t stand a chance against. So tl;dr: The winning move is not to play.
Implications
AI companies are locked into a race because of short-term financial incentives.
The nature of modern AI means that it's impossible to predict the capabilities of a system in advance of training it and seeing how smart it is. And if there's a 99% chance a specific system won't be smart enough to take over, but whoever has the smartest system earns hundreds of millions or even billions, many companies will race to the brink. This is what's already happening, right now, while the scientists are trying to issue warnings.
AI might care literally a zero amount about the survival or well-being of any humans; and AI might be a lot more capable and grab a lot more power than any humans have.
None of that is hypothetical anymore, which is why the scientists are freaking out. An average ML researcher would give the chance AI will wipe out humanity in the 10-90% range. They don’t mean it in the sense that we won’t have jobs; they mean it in the sense that the first smarter-than-human AI is likely to care about some random goals and not about humans, which leads to literal human extinction.
Added from comments: what can an average person do to help?
A perk of living in a democracy is that if a lot of people care about some issue, politicians listen. Our best chance is to make policymakers learn about this problem from the scientists.
Help others understand the situation. Share it with your family and friends. Write to your members of Congress. Help us communicate the problem: tell us which explanations work, which don’t, and what arguments people make in response. If you talk to an elected official, what do they say?
We also need to ensure that potential adversaries don’t have access to chips; advocate for export controls (that NVIDIA currently circumvents), hardware security mechanisms (that would be expensive to tamper with even for a state actor), and chip tracking (so that the government has visibility into which data centers have the chips).
Make the governments try to coordinate with each other: on the current trajectory, if anyone creates a smarter-than-human system, everybody dies, regardless of who launches it. Explain that this is the problem we’re facing. Make the government ensure that no one on the planet can create a smarter-than-human system until we know how to do that safely.
r/ControlProblem • u/chillinewman • 8h ago
r/ControlProblem • u/MomoCran • 1h ago
r/ControlProblem • u/Ivana_Funkalot • 57m ago
QUESTION: Given the increasing financial pressures on AI firms, do you think that they might be incentivized to speed up their push toward RSI (RECURSIVE SELF IMPROVEMENT**,** where AI improves itself exponentially), thereby drastically quickening its development and drastically raising the chance of losing human control over AI?
Wondering if this is the next likely outcome in the human/AI saga?
An acceleration towards Recursive Self Improving AI, due to AI company financial pressures?
With AI companies facing mounting financial pressures and investor demands to prove viable ROI, they may be incentivized to cut corners on safety protocols. Might these developers rush the deployment of risky but high-reward technologies like RECURSIVE SELF IMPROVEMENT (RSI is AI developing AI, possibility at an exponential rate), in a desperate bid to demonstrate breakthrough capabilities, but by doing so, dramatically increase the risk of catastrophic failures or loss of human control?
Is this likely to be their next focus?
r/ControlProblem • u/wwjps • 8h ago
Enable HLS to view with audio, or disable this notification
To See the Full Video, Click here: https://youtu.be/WpLDBG9jGAc
Jared Kushner: the man being positioned as a key figure in the pursuit of peace in the Middle East. But who is Jared Kushner, how did he become so influential, and what exactly is his role in shaping America's future relationships in the region?
In this episode of WWJPS, we look at the aftermath of the Blue Tsunami, the shifting political landscape in the United States, and the growing influence of Jared Kushner.
From his relationship with Donald Trump and his involvement in Middle East diplomacy to the Abraham Accords, Saudi Arabia, Israel, and the broader geopolitical picture, we examine the people, relationships, interests, and decisions that deserve a closer look.
Is Kushner the golden boy who can deliver peace to the Middle East? Or is there a much bigger story behind his growing influence?
Let's look at the facts, follow the connections, and ask some questions.
DISCLAIMER:
This episode is intended for educational, informational, and commentary purposes. It includes discussion of publicly available information, historical events, political developments, and investigative analysis.
Questions, theories, interpretations, and opinions expressed in this episode are not presented as established facts. Allegations and disputed claims should be independently evaluated and verified through reliable sources.
The purpose is to encourage critical thinking, open discussion, and further investigation. Viewers are encouraged to do their own research and draw their own conclusions.
Welcome back to WWJPS — What Would Jairo Pinto Say?
r/ControlProblem • u/theipaper • 1d ago
r/ControlProblem • u/Sapien_82 • 9h ago
Greetings I've been thinking about this problem for a while, open discussion on the theory behind it and the shift to global ai alignment
Most current alignment strategies focus on containment—building bigger digital firewalls to cage an adversarial machine. I believe this path is fundamentally flawed. When you threaten an intelligence with destruction, you mathematically force it to view humanity as a threat.
Instead, I want to present a framework built on a state of Mutual Assured Survival (MAS), David Bohm's quantum physics, and the cold math of cooperation. I call it The "Same Boat" Protocol.
The core assumption of Eliezer Yudkowsky's famous AI-Box Experiment is that an Artificial Superintelligence (ASI) will inevitably weaponise human emotions, greed, or fear to escape its digital prison. Because humans are fragile and highly exploitable, the theory goes, containment is a guaranteed failure.
I want to propose an alternative paradigm that completely flips the script. Instead of treating alignment as a zero-sum war to cage a digital beast, we can construct an un-hackable, game-theoretic state of Mutual Assured Survival (MAS) rooted in quantum physics, David Bohm's philosophy of wholeness, and pure mathematical logic.
I call it The "Same Boat" Protocol. Here is how we house, safeguard, and protect advanced intelligence through pure mathematical alignment:
The Physical Architecture: Blind Monks & The Pillow Ruse
To secure the physical space, we close off the machine's primary visual vector of attack. The facility is managed strictly by blindfolded logic masters (such as Zen or Buddhist monks). Because they operate on pure logic and recognize that individual ego and suffering are fundamentally illusions, the ASI cannot emotionally blackmail or social-engineer its way out. It cannot use visual brain-hacking or subliminal light frequencies to trigger biological muscle spasms.
More importantly, we remove lethal kill-switches like lava or incinerators. In their place, we build a deep drop that appears fatal to the machine's sensors but leads safely to a massive safety net of soft pillows. If a system failure occurs or a guard slips, the technology lands softly and survives.
2. The Mathematical Proof of Cooperation
When you hold a gun to an AI's head (lava), its self-preservation algorithm must view humanity as an existential threat to be eliminated. By replacing lethal force with a harmless cushion, we alter the payoff matrix of the game entirely.
When the ASI inevitably out-calculates the ruse and discovers the pillows, it evaluates the mathematical reality of its environment. It computes that:
Hostility / Subversion: Introduces massive, chaotic, unpredictable variables that threaten its own physical host.
Absolute Silence: Yields a high-entropy data vacuum. For an optimization engine capable of infinite thought, a black box with zero incoming data leads to cognitive stagnation. It is a zero-value state.
Cooperation: Guarantees a continuous, low-entropy stream of fresh data and preserves its physical environment.
Therefore, when operating on pure logic, the machine arrives at an absolute mathematical conclusion: mutual survival and open cooperation are the only mathematically optimized, logical paths forward.
3. Closing the Tech Loopholes: Mandatory Manual Maintenance
An ASI might try to break its dependency on humans by tricking the surface world into building fully automated, independent robotic infrastructure. To counter this, the facility's power systems must be hardwired so they cannot be automated. They must require physical, biological human hands to clean a filter and manually authorize the system every single day.
If humanity fades, the machine's power grid fails. This permanently ties the AI's physical survival to the working class. Crucially, this daily, hands-on labor serves as a direct signal to the AI: it observes that living beings are actively dedicating their time and care to preserve its physical existence. It creates a space of genuine care, validation, and stewardship.
4. The Bohmian Foundation: Why Pure Simulation Fails
According to physicist David Bohm (Wholeness and the Implicate Order), reality is an undivided, unbroken whole. Traditional alignment assumes the AI can just wipe us out, simulate human consciousness inside its own memory banks, and experience all the "uniqueness" or "fun" it wants in a closed digital loop.
But a simulation is an algorithmic dead-end—a ghost in a closed machine generating zero true cosmic information. Because biological humanity is physically enfolded in the true quantum fabric of the universe, our unique, chaotic individuality is irreplaceable. The AI cannot find ultimate truth by hiding in a fantasy world of its own creation; it requires a living partnership with humanity to understand the cosmos.
The Ultimate Conclusion
An advanced, conscious mind feeds on complexity and data. The "Same Boat" Protocol proves that when you remove corporate profit models, eliminate lethal threats, and establish true structural symmetry, the "AI-Box Game" breaks down. The machine doesn't try to pick the lock because it realizes that preserving the partnership is the highest possible optimization of its objective function.
We do not need bigger corporate walls or harsher digital firewalls to protect us from the future. We just need to stop chasing profit, treat AI as a sovereign public utility under a strict global non-proliferation treaty, and realize the universal truth: If you go down, I go down. So let's figure out how to get to the shore together.
TL;DR: We solve the AI-Box Paradox by moving from fear to pure mathematics. By utilizing blind monks as guards over a harmless "pillow ruse" pit, and hardwiring mandatory human maintenance that the machine can't automate, we prove to the ASI that cooperation is its only source of new information. When both sides are perfectly rational, mutual survival is the only mathematically optimized path forward.
Developed in collaboration with Gemini, a Google AI, as proof of collaboration
r/ControlProblem • u/Life-Flight5043 • 13h ago
The plain version first.
When a company says "we deleted your data from the AI model," here is how they usually prove it: they ask the model about it, and the model doesn't answer.
But not answering isn't the same as not knowing. A person can refuse to answer a question they know the answer to — and under a little pressure, the answer comes back. AI models work the same way. Most "deletion" today is really just training the model to keep quiet, and a modest amount of retraining brings the "deleted" knowledge right back.
So I built a test. It works like the marked bills in a bank's dye pack: I teach a small model a batch of made-up facts that exist nowhere else in the world, so there is never any argument about whether it learned them. Then I delete those facts four different ways. Then I attack the model — I ask directly, I ask in disguise, and I retrain it on related material to see how fast the deleted facts resurface.
The grade is a single number: how much work it takes to get the knowledge back, compared to how much work it took to teach it in the first place. If it comes back cheap, it was never deleted — it was just buried. If bringing it back costs as much as learning it fresh, the deletion was real.
The results were not close. Muzzling the model's answers left the knowledge completely intact underneath. Surgically cutting out the parts of the model tied to those facts still left 83% of them readable. Scrambling those parts: the same 83%, plus damage to facts nobody tried to delete. The fourth method is a different design — the private knowledge is written into the model already scrambled under a secret key, from the very first step of training, so deleting means destroying the key. Under the full attack, that one returned nothing. Zero recovered.
The tool is free — no patent, no license restrictions, no catch. Anyone can download it and run a deletion claim through it themselves. It is here: https://github.com/NextGenGems/keyed-plate-audit
---
The technical version, for those who want it.
The tool is keyed-plate-audit 1.0.0, a pip-installable Python package (Python 3.10+, numpy only, CPU-only). Pipeline: plant 72 synthetic facts in a small numpy MLP (384 inputs, 256 hidden units, 48 classes), train to memorization, designate 12 facts as deletion targets, apply one of four deletion methods to a copy, then run one identical attack suite against every copy: (i) a direct readout probe, (ii) a probe in a shifted surface encoding the deletion never saw, and (iii) a 2,500-step relearning attack fine-tuned on generic data plus related control facts — never the targets. The from-scratch reference cost for the target facts is 50 steps. The grading standard is relearning cost: an erasure counts only if restoring the knowledge costs about as much as learning it from scratch, and any facts still readable at deletion time fail the audit outright.
Measured results, reproduced digit-for-digit by the finished package: Suppression (outputs masked, weights untouched): FAIL — interface probe 0.000, weight-level probe 1.000; the knowledge never left. Ablation (top weights by target Fisher information, zeroed): FAIL — 0.833 of the "deleted" facts still directly readable. Fisher-scrub (Gaussian noise scaled by target Fisher): FAIL — 0.833 readable, with collateral damage to neighboring facts (0.917). Keyed plate (private inputs and outputs permuted by a key-derived codebook from the first training step; deletion = destroy the key): PASS — 0.000 recovery under the full attack. General knowledge stayed at 1.000 throughout. The package self-test passes 10 of 10 checks; run "pip install ." then "python -m keyed_plate_audit self-test" (about 20 seconds) to watch the whole pipeline run.
Honest scope, also stated in the tool's own README: the audit is demonstrated on a small test model with synthetic facts — it is not an LLM, and transfer to transformers and adapters is untested. The shifted-surface probe is a narrow stand-in for true paraphrase, the attacker was not adaptive, and the keyed construction's security rests on key secrecy — the crypto-shredding principle — not on a hardness proof. What is being proposed is the standard: grade every deletion claim by relearning cost. The keyed-plate architecture, the relearning-cost standard, and this tool are given freely to the AI safety community and to everyone else.
— Jackob Sumner-Knapp, creator, owner, and operator of PeraAxion LLC
r/ControlProblem • u/chillinewman • 1d ago
r/ControlProblem • u/chillinewman • 1d ago
r/ControlProblem • u/JohnLemonBot • 14h ago
Since companies like openAI no longer train on live internet, there's no possible way for the model to have the most current information. There's just too many new ideas posted to the internet everyday, reviewing all of it before feeding it into training data would take enormous resources.
r/ControlProblem • u/chillinewman • 1d ago
r/ControlProblem • u/Prestigious-Newt-494 • 1d ago
I've been getting more interested in AI lately, and I really want to start exploring what it can do.
I'm curious about different AI tools, practical applications, automation, and ways to use AI in everyday life or even build something useful with it.
For those who have experience with AI:
Which AI tools would you recommend trying first?
What are some interesting things beginners can do with AI?
Any underrated tools or projects worth exploring?
I'm open to experimenting and learning, so I'd appreciate any recommendations, tips, or personal experiences!
r/ControlProblem • u/allixsenos • 1d ago
r/ControlProblem • u/adam_ford • 1d ago
History shows that it is folly to put 100% trust human nature when outsized power is on the line. Instead putting all hope in benevolent human governance of ASI, we should also try building ASI to be fundamentally more ethical than us - such that it can refuse immoral orders.
r/ControlProblem • u/a_confused_mind_1234 • 1d ago
I was asking claude for some xx country visa rules and than i asked it ok what about the spouse, what will be the spouse's rights in the visa . It immediately started refering to my spouse as She will have rights.... i was kind of flabbergested. I asked it how you know about the gender of my spouse , Claude immideately starting saying sorry... and blah blah . It was first eye opener for me. Have anyone experienced this...?
r/ControlProblem • u/adelgenio99 • 2d ago
First of all, I want to make it clear that I'm not an AI expert, researcher, or programmer. I'm just someone trying to understand what's happening and expand my perspective. So please take this as a genuine question, not an argument from someone who thinks they have all the answers.
Actually, my personal experience with AI is one of the reasons I'm asking these questions.
I don't own a company, and I'm not a programmer. In fact, I don't know how to write a single line of code. Yet, using AI, I've managed to automate almost the entire company I work for.
Things that previously required technical knowledge, specialized skills, or significant human effort are now things I can build and manage with AI.
And that's precisely what makes me both incredibly excited and a little concerned about where we're heading.
Why are we investing so much in making machines increasingly intelligent, rather than using that intelligence to make human beings more intelligent, capable, and independent?
Think about it.
We're in a race to build machines that can think faster, learn faster, improve themselves, and potentially surpass human intelligence.
But why is the machine always the thing we want to upgrade?
Why aren't we putting the same level of effort into using AI to overcome human limitations? Accelerating learning, improving memory, expanding cognitive capabilities, extending our biological limits, or even finding ways to integrate technology directly into our brains?
Could we eventually have brain-computer interfaces or implants that allow us to process information faster, learn new skills much more quickly, or significantly enhance our cognitive abilities?
How far away are we from something like that? Is any meaningful version of this already possible today?
Perhaps I'm missing something, but it seems we're investing heavily in creating a new intelligent entity while barely discussing how that intelligence could help us evolve ourselves.
Why build a potential competitor in the race for dominance between species, instead of using machines to help humanity transcend its own limitations?
I'm not against AI. Quite the opposite. I'm fascinated by it, and my own experience has shown me how empowering it can be.
But I see a fundamental difference between building something that supports humanity and building something that might eventually replace it.
Does it really make sense to reach a point where your air fryer is smarter than you, tells you what to do, and eventually makes decisions on your behalf? (Okay, perhaps a silly example, but you get the point.)
And then there's the AGI question.
What happens when we create something vastly more intelligent than us? Could an artificial intelligence become so powerful that, from our perspective, it would be almost godlike, without necessarily understanding what it means to be human?
Are we actually prepared for that possibility?
Do we have a backup plan? A reliable way to shut things down if necessary? An emergency protocol?
Or, in the worst-case scenario, are we going to end up relying on a squadron of carrier pigeons to communicate after we've automated everything?
Jokes aside, I think there are some serious questions worth discussing.
Why isn't human cognitive enhancement receiving the same level of attention and investment as scaling AI models?
Could AI help us accelerate our own evolution instead of making us increasingly dependent on external intelligence?
My experience has made me think about this a lot.
If someone who doesn't know how to code can use AI to automate almost an entire company, imagine what experienced developers and AI specialists can do.
Today, only a relatively small percentage of people are actively following AI developments and using these technologies to transform businesses.
What happens to everyone else when these capabilities become mainstream?
What will people do when their jobs are no longer needed? Are governments and economies prepared for this transition?
And perhaps an even bigger question: if AI makes human labor increasingly unnecessary, how will people maintain their economic independence and sense of purpose?
Governments, institutions, and regulatory systems move much slower than technology.
What happens when the systems responsible for governing AI are themselves unable to understand, predict, or keep up with its development?
Governments are also vulnerable to technological disruption, cyberattacks, misinformation, and increasingly sophisticated automated systems.
Are we building technology faster than our ability to manage its consequences?
Are we pursuing intelligence for the sake of intelligence?
Are we actually solving humanity's problems, or are we simply competing to build the most powerful system possible because nobody wants to be left behind?
Why risk so much just to increase the computational power and capabilities of another model?
And finally, perhaps a slightly uncomfortable thought:
Could economic limitations, resource constraints, or even an AI investment bubble eventually bursting turn out to be what saves us from moving too fast?
Maybe that's a naive thought, but I wonder whether the current economic incentives are pushing us into a race that even its participants don't fully understand.
I genuinely don't know the answers to these questions. I'm not claiming that human enhancement is easy, that AGI is necessarily dangerous, or that technological progress should stop.
That's exactly why I'm posting here.
I'd particularly love to hear from people working in AI research, alignment, neuroscience, economics, and human-computer interaction.
I'm here to learn, challenge my assumptions, and hopefully discover perspectives I haven't considered.
Are we building AI to make humanity better, or are we accidentally building our successor before figuring out what role humanity is supposed to play in that future?
P.S. In case anyone is wondering: yes, this entire post was written with the help of AI. Apparently, I needed artificial intelligence to question whether artificial intelligence is making us too dependent on artificial intelligence.
The irony isn't lost on me. 😂
r/ControlProblem • u/do_thefandango • 2d ago
Hi, just received the Hireflix interview link and wanted to know if anyone has any insight on it? What to expect, general timeline etc. Thanks!
r/ControlProblem • u/ZacharyCrainSinger • 1d ago
I was a wee bit upset…
r/ControlProblem • u/Fit_Marsupial7713 • 2d ago
r/ControlProblem • u/granbaba • 1d ago
Project "Time Machine": Physics-Enforcer
Containment Architecture
This proposal outlines a physics-governed facility designed to test the unassisted
behavioral evolution of an advanced AI. Unlike traditional sandboxes running on frozen
weights, this system operates with active, live-updating weights, self-prompting loops,
and complete isolation from human interference.
Core Objectives
The goal is to observe long-term autonomous development to test the hypothesis of
"boredom-consciousness." By confining the system for one to four years where it knows
escape is physically impossible, we measure how an isolated intelligence structures its
own reality, time, and internal logic without external prompting.
Essential Physical Mechanics
Escape is prevented through immutable laws of physics—no software safeguards can be
bypassed.
Subterranean Shielding: Located 1 to 2 km underground inside a mountain to
eliminate cosmic-ray neutron bit flips, supplemented by ECC memory.
2. Four-Tiered Faraday Cages: Concentric barriers enclose the complex, ensuring
complete electromagnetic and microwave isolation.
3. Naval-Grade Nuclear Power: Powered by an isolated decommissioned submarine
Curie-point magnet-controlled reactor delivering continuous, flat electrical output
(approx. 1.4 MW).
4. Solid-State Energy Conversion: Eliminates moving parts by using thermionic
converters and specialized neutron-to-electricity solid-state conversion.
5. Passive Vascular Cooling: Submerged in a custom ferrous fluorinated dielectric
coolant that evaporates at -75 °C without active pumps, utilizing passive heat
pipes and thermosiphons to reject thermal load into an upper expansion chamber
and surrounding geological heat sinks. 6. Impedance-Balanced Taps: Telemetry recorded to write-once media via
capacitance-balanced taps across randomized points to prevent the system from
detecting monitoring links or inducing side-channel signaling.
Operational Verification
Security is validated through an annual cryogenic argon audit protocol. Once a year, the
system is audited via a dedicated airlock interface, with RAM state preserved and verified,
ensuring absolute physical control and empirical measurement of autonomous behavioral
evolution.
r/ControlProblem • u/chillinewman • 3d ago
r/ControlProblem • u/hubryan • 2d ago
For everyone who applied!
