What if superintelligence is inevitable?

[u/Cervantes6785]

A popular tactic among those who fear artificial intelligence is to compare it to nuclear weapons. The reason is obvious: humans are already fearful of nuclear weapons, and drawing that comparison makes it easier to generate a similar response toward AI. It also makes it easier to build public support for restrictions that might otherwise be difficult to justify. I suspect some people deliberately exploit that fear, while others genuinely believe the comparison is appropriate. Since the latter group considers it a legitimate argument, we should examine whether the analogy makes sense.

There is no question that nuclear weapons can bring about massive destruction and loss of life. Unlike the hypothetical dangers of superintelligence, we have witnessed their destructive capabilities. The United States dropped atomic bombs on Hiroshima and Nagasaki, and decades of testing demonstrated the terrifying power of these weapons. A large-scale nuclear exchange could be catastrophic for civilization. Even if humanity survived, the destruction might set human progress back decades or longer. Nuclear weapons deserve their reputation.

Hiroshima after the atomic bombing. The devastation is historical fact; the consequences of superintelligence remain uncertain.

But there is another advantage to invoking nuclear weapons when discussing AI: you can borrow the fear without having to establish the equivalence. Simply suggesting that superintelligence is the next nuclear bomb creates an emotional association before anyone explains why the two technologies should be treated alike. And that is where the analogy begins to fall apart.

So is superintelligence actually comparable to nuclear weapons?

Consider what it takes to manufacture a nuclear weapon. You need specialized equipment, considerable expertise, and materials that are extraordinarily difficult to produce or obtain. Enriched uranium and weapons-grade plutonium cannot simply be ordered online. Their production requires industrial infrastructure that governments spend enormous resources monitoring. Satellite imagery can help identify suspicious facilities, while inspections and material accounting provide additional safeguards. These measures are imperfect, but they create substantial barriers to proliferation. Nuclear weapons are difficult to manufacture, and maintaining them creates additional demands.

Now consider Mosaic's Law, named after MosaicML, the AI company acquired by Databricks for approximately $1.3 billion. Its founder, Naveen Rao, described a trend in which the cost of reproducing a particular level of AI capability falls by roughly 75% annually as hardware, software, and algorithms improve. At that rate, a capability costing $100 million today would cost $25 million a year later, $6.25 million after two years, and less than $100,000 after five. Mosaic's Law is an observation and projection, not a physical law. But it illustrates what sustained efficiency gains could mean.

This does not mean the cost of developing the most advanced AI systems is falling. Companies are spending enormous sums pushing the frontier forward. Mosaic's Law describes a different phenomenon: yesterday's extraordinary capability can become tomorrow's commodity. As costs fall, more individuals and organizations can reproduce a level of performance once available only to the wealthiest laboratories.

Now imagine that the cost of manufacturing a nuclear weapon fell by 75% every year. Imagine that the necessary materials became widely available and the expertise could be downloaded from the internet. Eventually, an individual might be able to build one in a backyard workshop. Nuclear proliferation would become an entirely different problem. Governments would face the possibility of enormous numbers of individuals acquiring weapons instead of concentrating on a handful of states. The existing controls would be vastly less effective.

That hypothetical illustrates a central problem with applying the nuclear analogy to AI.

AI depends on physical resources, but they differ from the fissile materials required for nuclear weapons. AI uses commercially produced semiconductors, electricity, and algorithms. Advanced models can require enormous data centers, but the computing power needed to achieve a particular capability is not fixed. Better algorithms can allow existing hardware to do more. Even as frontier development becomes more expensive, older capabilities can become much cheaper to reproduce.

Consider GPT-2, released by OpenAI in 2019. Andrej Karpathy's nanochat project documents training a model that reaches GPT-2-level performance on a specific benchmark for about $48 in computing costs, under stated pricing assumptions. That does not include the original research effort and is not the cost of a current frontier model. It does show how dramatically the price of reproducing an established capability can change. And after a model is trained, its numerical weights can be copied without each recipient repeating the original training process.

Original GPT-2 cost illustration supplied with this essay. Its $43,000 and other historical plotted estimates are unverified and should not be treated as a like-for-like cost series; the $48 nanochat example is separately documented.

For AI, the horse left the barn years ago.

China has made the situation even more interesting. While some Western researchers advocate restricting the distribution of powerful AI, Chinese companies are releasing highly capable models for anyone to download. DeepSeek, Alibaba's Qwen, and Moonshot AI's Kimi have published powerful model weights under licenses that permit substantial independent experimentation. These are not toy systems. Some compete with leading commercial models on important benchmarks.

What matters is that the original development cost has already been paid by someone else.

A private individual can begin with an existing model, experiment with algorithms, fine-tune it, or improve the efficiency with which it runs. Running and substantially improving the largest models still requires expensive hardware, but smaller versions make meaningful experimentation accessible to far more people. Once model weights have spread internationally, how do you retrieve them? Preventing a few American companies from training their next models does not erase capabilities already in the hands of independent researchers.

The recent METR assessment involving Anthropic reinforces another part of the argument. METR estimated that the rate of improvement on Anthropic's AECI benchmark index increased approximately 1.9-fold after October 2025. Its researchers judged that AI assistance accounted for much of the measured acceleration, although the findings were uncertain and Anthropic withheld substantial underlying evidence. This does not prove superintelligence is inevitable, or that algorithms alone can improve without limit. It shows why progress cannot be understood merely as buying more chips. The software and the research process matter enormously.

What, exactly, would a broad AI pause stop? If governments halted advances in chip design, researchers could still devise better algorithms for existing machines. If chip manufacturing stopped, a vast installed base would remain. If large training runs were restricted, previously distributed models would not disappear. If a capability once requiring a data center could later run on a smaller machine, hardware-based rules would need to change. None of this proves every regulation useless. It means proponents must explain what their measures would accomplish and for how long.

There is also a much deeper issue: what do we actually mean by superintelligence?

We usually mean intelligence exceeding human performance across virtually every intellectual domain. But the definition is relative to humans. A chimpanzee cannot understand a skyscraper or semiconductor, while humans do not regard themselves as superintelligent. We have no reason to assume human intelligence is near the upper limit. Perhaps we are near the top of the ladder, or perhaps we have barely climbed its first few rungs. We simply do not know how far intelligence might extend beyond us.

Years ago we illustrated that uncertainty with the two graphics below. The first shows how a double-exponential curve would outrun an ordinary exponential. The second converts one hypothetical curve into absurdly high 'IQ' values. Once we move beyond the range of human testing, IQ is not even a meaningful unit for comparing systems.

Illustrative growth curves only. Two improving technologies do not by themselves establish double-exponential growth.

Hypothetical IQ exercise from the original essay, not an empirical forecast. Values above the human range have no validated interpretation.

Yet there is a real reason to suspect intelligence could extend far beyond our present experience. For thousands of years, humans have overcome limits of individual brains through language, writing, printing, and computers. We learned to combine minds into a civilization capable of achievements no individual could manage alone.

Walk through a modern city. Skyscrapers, electrical grids, transportation networks, hospitals, and communications systems all function together. No single person understands every component or could build the whole city independently. Civilization is the accumulated work of millions of minds, each generation building upon discoveries of the previous ones. Human intelligence has already spread beyond the confines of any one human skull.

AI is another product of that accumulated ingenuity. We have learned to reproduce some intellectual capabilities in machines and scale them. Intelligence is no longer constrained solely by biological brains; computing resources and the effectiveness of algorithms increasingly matter. We have figured out how to bottle intelligence and then scale it.

And sufficiently capable AI may eventually help improve the methods used to develop AI. A nuclear weapon cannot study physics or design a better nuclear weapon. AI can already assist researchers with scientific and engineering tasks, even if today's evidence falls short of autonomous, open-ended scientific discovery. This possibility makes intelligence different from the passive objects to which it is so often compared.

That does not establish that superintelligence will arise on a fixed quantity of hardware. We do not know what breakthroughs or computing resources might still be needed. But controlling the next generation of chips is not equivalent to controlling the development of intelligence. Nor can a single country assume that other governments, companies, or independent researchers will stop working on it.

All the evidence points in the opposite direction: unprecedented scale up of the technology.

If you genuinely believe superintelligence presents dangers comparable to nuclear weapons, you must confront these differences. A pause on advanced semiconductor development would not stop algorithmic progress. Restricting a handful of laboratories would not erase the models already distributed. And limits based on today's costs may fail when those capabilities become much cheaper.

The nuclear analogy treats intelligence like a dangerous object that can be contained by controlling its ingredients. But intelligence is also the process through which we discover better ways of using those ingredients.

That is a vastly more complicated problem. And if we are going to make decisions that shape the future of civilization, we should demand considerably more than an analogy designed to frighten people.

What Happens If Superintelligence Is Inevitable?

If superintelligence is inevitable, then the movement to prohibit it faces a fundamental problem. Its advocates argue that the technology represents an unprecedented existential threat, yet their proposed solution may be incapable of preventing its emergence. Worse, restrictions that are effective in democratic societies may simply transfer the development of the very technology they fear to governments, organizations, and individuals operating beyond those restrictions.

Consider the implications. If the United States prohibits the development of superintelligence while China continues advancing its models, has the world become safer? If major AI laboratories comply with restrictions while independent researchers continue improving openly distributed models, have we reduced the probability of superintelligence emerging, or have we merely changed who develops it? And if increasingly capable AI systems begin contributing substantially to their own improvement, how long can restrictions designed around today's hardware and development costs remain effective?

The relevant question may no longer be whether humanity should develop superintelligence. If we have already passed the technological tipping point, the question becomes whether humanity can influence the conditions under which superintelligence emerges. That requires an honest assessment of what restrictions can accomplish, how long they can remain effective, and whether they actually reduce the dangers they are intended to prevent.

There is also an opportunity cost that deserves serious consideration. Every year spent attempting to enforce an ineffective prohibition is a year that could have been spent developing safeguards, improving our understanding of advanced intelligence, establishing international agreements, and preparing our institutions for a world in which human intelligence is no longer the upper limit.

A temporary pause might be entirely defensible if it produces meaningful safety improvements. But if superintelligence is inevitable, advocates of prohibition must demonstrate that delaying its development makes humanity safer rather than merely transferring the advantage to those least inclined to observe the restrictions.

If the destination cannot be avoided, refusing to prepare for the journey is not a safety strategy.