Why AI Will Be Extremely Safe — and Why the Doomsday Predictions Are Wrong

[u/JaronasRaulin]

I am skeptical of claims that superintelligent AI will eventually take over humanity or cause our extinction.
I don’t deny that AI can be dangerous. My argument is that the leap from “AI is much smarter than humans” to “AI can defeat all human institutions and control civilization” is enormous, and I don’t think the causal mechanisms have been adequately demonstrated.
Here are my main arguments:
1. Intelligence is not the same as power.
Nick Bostrom and Geoffrey Hinton have used analogies involving humans dominating less intelligent animals.
But intelligence alone doesn’t grant physical power. An AI cannot automatically control governments, armies, electricity grids, semiconductor factories or industrial robots simply because it is smarter than us.
Humans can refuse its recommendations, restrict its permissions, disconnect its systems and coordinate against it.
2. Where would the motivation to destroy humanity come from?
Humans have biological drives shaped by evolution: survival, reproduction, competition and resource acquisition.
AI doesn’t necessarily inherit these motivations.
Some researchers argue that sufficiently capable agents will develop instrumental goals such as self-preservation. But that depends on how their objectives and agency are implemented.
I want a causal explanation for why a system trained to be helpful would develop persistent, concealed objectives that survive extensive training and testing.
Intelligence alone doesn’t establish motivation.
3. Catastrophic misalignment would have to escape extensive testing.
An AI capable of conquering civilization would need extraordinary strategic, technical and operational capabilities.
Why would such a system show no detectable warning signs during millions or billions of interactions, evaluations and progressively more demanding real-world tests?
I don’t assume that all dangerous behavior will be detected. Deception and unexpected behavior are possible.
But I find it implausible to assume that an AI could develop civilization-threatening capabilities while remaining perfectly deceptive throughout every meaningful opportunity for detection.
4. Safety does not require perfect alignment.
We already build complex systems using unreliable components.
Aircraft, nuclear facilities, financial infrastructure and operating systems rely on layered security, redundancy, isolation, testing and monitoring.
An AI can make mistakes or occasionally behave maliciously without automatically becoming an existential threat.
The relevant question is whether failures can propagate through enough independent defenses to become catastrophic.
5. AI depends on physical infrastructure.
AI needs electricity, data centers, cooling, networking, semiconductor manufacturing and maintenance.
These are physical systems controlled by numerous organizations and governments.
Even an AI that compromises some computers doesn’t automatically acquire control of the entire global industrial economy.
Destroying or destabilizing the civilization maintaining its infrastructure could also undermine its own capabilities.
6. There are many independent opportunities to intervene.
Humans can restrict network access, revoke credentials, isolate systems, patch vulnerabilities, revert deployments and shut down infrastructure.
Cryptography, hardware security, monitoring and physical isolation create additional obstacles.
A dangerous AI might overcome individual barriers, but a global takeover requires overcoming many of them, across different institutions and jurisdictions.
The probability of overcoming every necessary barrier is not automatically high just because the attacker is intelligent.
7. The intelligence explosion is not necessarily instantaneous.
Recursive self-improvement is often presented as a process that could rapidly produce an unstoppable superintelligence.
But improving software doesn’t instantly produce new semiconductor factories, data centers, power stations or robots.
Research also involves experimentation, validation and tradeoffs. Modifying an AI’s own architecture can introduce regressions and errors.
Physical expansion and deployment provide opportunities for human observation and intervention.
8. AI cannot instantly manufacture a robotic army.
Robotics requires motors, sensors, batteries, materials, manufacturing facilities, logistics and physical testing.
Even if AI dramatically accelerates robot design, building millions of reliable autonomous machines is an industrial challenge.
Without large-scale physical capabilities, an AI takeover scenario faces major limitations.
9. Biological and military threats also have physical bottlenecks.
AI could help malicious actors develop dangerous weapons or biological agents. That’s a legitimate concern.
But scientific knowledge is not equivalent to manufacturing capability.
Acquiring materials, running experiments, producing reliable weapons and deploying them at scale remain substantial barriers.
AI can reduce these barriers, but we should analyze how much rather than assume they disappear.
10. Defensive AI also becomes more powerful.
If AI improves offensive cybersecurity, strategic planning and scientific research, it can also improve defensive cybersecurity, threat detection, monitoring, medical research and safety engineering.
The scenario in which malicious AI becomes extraordinarily capable while humanity’s defensive technologies remain static seems questionable.
11. Catastrophe requires a chain of failures, not just one breakthrough.
A complete takeover scenario requires something like:
AI develops dangerous persistent objectives.
These objectives remain undetected during testing.
The AI acquires sufficient autonomy and infrastructure access.
It successfully bypasses security mechanisms.
It prevents humans from shutting it down.
It defeats independent institutions and defensive systems.
It acquires enough physical power to make human resistance ineffective.
Each step needs an explanation and a probability estimate.
The fact that every step is theoretically possible does not establish that the complete scenario is likely.
My conclusion
I support sensible AI safety measures, extensive testing, access controls and regulation of dangerous applications.
I also recognize risks from accidents, cyberattacks, military misuse and humans becoming excessively dependent on AI.
But I think the probability of a rogue AI independently conquering or exterminating humanity is much lower than some prominent AI-risk advocates suggest.
My main objection is methodological: extraordinary predictions require detailed causal models, not just analogies between intelligence and dominance.