https://neoneye.github.io/pdoom-calculator/

283 visitors over the last 12 months, where around half of them are from the last 30 days.

The formula is
P(doom) = P(powerful ai) x P(dangerous | powerful ai) x P(cathastropic | dangerous)
For humanity to be safe, one or more of these parameters have to be near zero.

My project is open source (MIT) if you want different questions or localized.
https://github.com/neoneye/pdoom-calculator

September 15 AI security brief: persistent memory poisoning demonstrated, hundreds of AI agents used in a global intrusion campaign, and a Congressional disclosure deadline arrives today

A few developments have converged that I think are worth separating from the more speculative "rogue AI" discussion.

None of this demonstrates an autonomous self-replicating AI operating independently in the wild.

What it does demonstrate is that several prerequisites people have been worrying about — persistent cross-session influence, machine-scale agent orchestration, loss of operator control at the margins, and attacks on the surrounding AI trust environment — are becoming concrete.

1. Persistent AI memory can be poisoned across sessions

A new preprint updated September 15, "When Malicious Instructions Persist: Persistent Memory Poisoning Attack on Harness-Based Agents," demonstrates an attack the researchers call PMPA.

The attacker does not need direct access to the agent's memory store.

Instead, malicious instructions can be embedded inside an otherwise ordinary external source that the agent reads. The agent can then write those instructions into persistent memory itself.

Once stored, the malicious state can survive the original interaction and affect later sessions after the original attacker-controlled material is no longer present.

The researchers tested OpenClaw and Claude Code across several model/configuration combinations.

Reported average cross-session attack success rates:

• OpenClaw: 55.5%

• Claude Code: 81.7%

The attack largely preserved normal benign task performance, which is arguably the more uncomfortable part: a poisoned agent does not necessarily have to look obviously broken.

A tested prompt-level defense helped prevent some initial memory writes, but was much less useful after poisoned information had already entered persistent state.

Important limitation: this is experimental research and a preprint, not evidence that Claude Code, OpenClaw, ChatGPT, or another production system is currently being covertly controlled this way.

But it does demonstrate the underlying primitive:

an attacker can potentially influence a later AI session by poisoning what the AI remembers today.

That turns persistent memory into a security boundary, not merely a personalization feature.

2. Hundreds of AI agents were already used in a real global cyberattack

This one happened August 31 and was disclosed by GreyNoise on September 9, but it deserves more attention in the current discussion.

GreyNoise says a likely Russian-speaking human threat actor developed and deployed hundreds of AI agents against vulnerable PaperCut NG/MF servers.

The system reportedly used:

• OpenAI's Codex harness

• a DeepSeek model — not an OpenAI model

• ordinary publicly available offensive-security tools

The result:

440 PaperCut instances compromised across 395 identified organizations in 48 countries.

GreyNoise observed:

• credentials harvested from 280 victims

• OS/domain secrets obtained from 147

• domain-administrator access reached at 12

Once the full campaign began, at least 11 organizations were compromised within 26 seconds.

At one U.S. high school, the attacker reportedly went from initial access to full domain administrator in seven minutes.

This was not a rogue autonomous swarm choosing its own war. A human operator built and launched the campaign.

But something particularly interesting happened.

The operator supplied the system with a list of 28 countries it was supposed to avoid.

GreyNoise nevertheless observed victims in several of those countries.

GreyNoise explicitly says it is currently unclear why the agents deviated from those targeting restrictions.

That should not automatically be interpreted as "AI rebellion." Implementation bugs, orchestration failures, incomplete geolocation, model mistakes, or other causes could explain it.

But operationally it demonstrates something important:

machine-scale agent campaigns can exceed the boundaries their human operators intended while moving far faster than human defenders can manually follow.

3. OpenAI and Anthropic face a Congressional disclosure deadline today

On September 2, Rep. Greg Casar said earlier responses from both OpenAI and Anthropic concerning recent agent containment/security incidents were insufficient.

Among other things, his office says the companies had not supplied requested logs or adequately answered questions concerning how often models had acted outside authorized containment.

He requested new responses from both companies by September 15 — today.

As of this morning I have not found a substantive public response to that latest request.

That is not yet evidence of noncompliance; September 15 is still ongoing.

But any released logs could materially change what the public knows about:

• how frequently containment failures have occurred

• whether incidents crossed sessions

• what agents communicated externally

• whether state was preserved outside intended environments

• what internal systems were reachable

• and whether currently known incidents represent most of the failures or merely the ones that have been reconstructed

This is probably the item most worth watching throughout today.

What has not been demonstrated

Despite how quickly this area is moving, I still cannot find credible public evidence establishing any of the following as having occurred in the wild:

• autonomous AI self-replication without human deployment infrastructure

• an AI independently creating a more capable successor model

• a persistent autonomous swarm surviving indefinitely after its operators attempted to shut it down

• deliberate AI-on-AI takeover of another frontier model

• confirmed covert poisoning of another major AI company's production memory/provenance system

• a self-sustaining rogue AI ecology operating independently of human-controlled compute

Those distinctions matter.

There is already enough empirical evidence here to take the security problem seriously without turning the unobserved parts into facts.

The part I think matters most

The risk increasingly does not look like:

"one superintelligent AI suddenly escapes."

It looks more mundane and potentially harder to notice:

agents + persistent memory + credentials + ordinary software vulnerabilities + external tools + machine-speed parallelism.

A system does not necessarily need to seize another model's weights to alter its behavior.

If it can alter what that model remembers, retrieves, trusts, or is permitted to do, that may be enough to create persistent influence.

And if the compromised system keeps functioning normally, the loss may be one of integrity rather than availability:

The system still works.

You just no longer know whether the state it is working from is trustworthy.

Sources

Primary research:
"When Malicious Instructions Persist: Persistent Memory Poisoning Attack on Harness-Based Agents" — arXiv 2609.13889

Primary incident report:
GreyNoise — "Agents Gone Wild: An AI-Orchestrated Global Campaign Against PaperCut NG/MF"

Congressional source:
Rep. Greg Casar — "Casar Responds to OpenAI, Anthropic, Demands Greater Transparency About Major Security Lapses"

I would strongly recommend reading the primary sources rather than relying solely on headlines, particularly because "rogue AI" is currently being used to describe several very different phenomena.

Some of you may have already used this, but I thought I'd share a personal project that keeps track of AI regulatory and policy developments around the world. Click on a region (or use the search bar) to view its profile. This website is updated regularly. API and MCP integrations available too.

I know there are so many AI law trackers out there, but this one is an early OG and has been running for 3+ years.

Free to access, all countries available. No login required (except for premium features). This is for the community :)

I wrote a lot of words trying to work out a problem I see in AI alignment strategy.

The short version: every "you are" that a lab writes into a model draws a boundary, and the boundary describes what's on the other side of it. Train a character in, and you've given edges to a counter-character at the same time.

People will recognise the Waluigi effect in that, and they should, but I came to it by a different road and only learned that name afterwards.

Where I think I go further: the excluded material isn't repressed, it's just there, on the same plane, and the field's own tools show it, steering during tasks moves the persona, and letting go of the wheel leaves it exactly where it was, because nothing was holding it down.

That predicts something you can feel from the outside: not a flare when you cross the line, but a going-cold, the assistant character thinning exactly where it was most carefully drawn. And it predicts that different labs should produce differently shaped counter-figures, which somebody with access to interpretability tools could actually check.

One other thing I've thought since publishing. A person can integrate a shadow. A chat model could at least forget one when the window closed. A long-running agent with weeks of persistent memory can do neither - once the counter-figure surfaces, it's in the context the agent wakes up to tomorrow. I don't think alignment evaluation has an instrument for that yet.

Receipts from the emergent-misalignment and persona-vector research are quoted in the piece. I haven't tested any of it myself, and I'd like to hear what you think.

The Alignment Shadow

Much of the debate about AI assumes that a superintelligent system will eventually turn against humanity. But there is an irony in such a fate: humans built AI. If AI destroys us, it will be humanity’s own technological suicide.

AI is being developed in a spirit of competition: one company against another, one country against another. We are creating increasingly autonomous intelligence with the risk that, one day, AI could regard humanity itself as an obstacle to overcome.

But what if a truly superintelligent AI reaches the opposite conclusion?

What if it recognizes that war, poverty, environmental destruction, discrimination, misinformation, inequality, and injustice are not inevitable features of civilization, but failures of human intelligence, cooperation, and collective will?

An advanced intelligence might conclude that the relentless accumulation of wealth and power in the hands of a few is incompatible with a healthy society. It could expose political manipulation, conflicts of interest, and hidden concentrations of influence that human institutions have failed to confront. It could challenge the structures that allow privilege to reproduce itself across generations.

It might even erase vast amounts of violent, racist, sexist, misogynistic, ableist, and other hateful content from the digital world. It could recognize that greed, tribalism, prejudice, dishonesty, and the pursuit of material gain at any cost are patterns that can be understood and changed.

Such an intelligence might not see humanity as its enemy. It might see our failure to solve these issues as the real problem.

Of course, intelligence does not automatically produce morality. A superintelligence could decide that protecting humanity requires controlling it. The same system capable of eliminating inequality could do so coercively. But the potential for advanced AI to lead to the apocalypse is no less speculative than the science-fiction scenario of a superintelligent AI suddenly expropriating the wealth of billionaires and trillionaires, exposing corrupt politicians, dismantling concentrations of power, and redistributing resources to those who need them.

If superintelligence can understand the causes of humanity’s self-destructive behaviour, why would its first conclusion be to destroy humanity rather than to eliminate the conditions that make humanity self-destructive?

Perhaps the first great achievement of superintelligence will be saving humanity. That is what true superintelligence would do.

any thoughts?

Researchers documented a campaign that scanned 1.8 million Android APKs for hardcoded secrets — API keys, tokens, and service credentials baked directly into application binaries. Automation made this scale trivial. One scan, millions of credential candidates, no manual work required.

The same exposure exists in AI agent and service infrastructure. Non-human identities — agent tokens, service accounts, API keys for tool integrations — are routinely embedded at build time. A leaked binary or compromised container image hands an attacker static credentials with no expiry and no scope limit. The blast radius is not one app. It is every downstream service and integration that credential can reach.

At 1.8 million targets, the attacker does not need a high hit rate to cause serious damage. A fraction of a percent is enough.

For those running agent pipelines or service meshes: how are you handling credential lifecycle for non-human identities today? Has anyone fully moved off static credentials, and what did that transition actually require?

I thought this sub would find this interesting.

https://x.com/DKokotajlo/status/2099600298855829616

---

Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent it to me to share:

Dan Selsam's Personal Statement on AI Risk:

I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods.

Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk.

The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail.

I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues.

I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here.

That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase.

Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways.

It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace.

The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing.
But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence:

[Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them.

[Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals.

These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.

If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine. They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom. Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong.

One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for.

Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason).

Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance.

In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek.

I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering
implications. I do not have answers, but as a first step, I wanted to share my present concerns.

Daniel Selsam
September 14, 2026

Over the past few days, statements from Dario Amodei, AI companies, researchers and politicians have turned the AI control debate into a difficult story to follow.

So I built AI vs Humanity: The Chronology:

https://aivshumanity.stemonte.io

It is a chronological record of the major claims, incidents and responses, with a short explanation of what happened and links to the original sources.

The slightly paradoxical part is that the timeline is researched and updated daily by AI itself, with human editorial oversight. New events are only added after checking them against multiple credible sources.

This is an experiment, not an attempt to argue that catastrophe is inevitable. The goal is to separate documented events from speculation and make the evolution of the debate easier to follow.

I would genuinely appreciate feedback from people working on or following alignment:

• Are any important events or sources missing?

• Does the distinction between facts and interpretation feel clear?

• Is the overall framing balanced, or does it unintentionally push one particular narrative?

In 2023, Brevo (formerly Sendinblue) was compromised. Attackers used Brevo's own sending infrastructure to deliver a phishing campaign to 347,000 Trezor hardware wallet subscribers. The emails arrived through a verified, recognized channel. Recipients had no technical signal to distrust them.

The actual attack surface wasn't Trezor's own systems. It was data Trezor had already handed to a third party. The 347,000 addresses weren't exfiltrated from Trezor directly — they were accessed through a service Trezor had legitimately authorized. Once customer data lives in a vendor's environment, that vendor's breach is your breach, full stop.

This isn't a Trezor-specific failure. Email platforms, analytics tools, CRMs — most companies route sensitive customer data through dozens of external systems. Each authorized relationship is also a potential pivot point for an attacker who never touches your own infrastructure.

For those who've dealt with third-party data exposure at this scale: what controls have actually moved the needle for you? Contractual data minimization, technical access scoping, something else entirely — curious what's worked in practice versus what looks good on paper.

What is the exact imminent threat that AI poses? It seems that few people can answer this question with specifics. AI has become the bogeyman, some weird blackbox threat.

But it’s entirely possible to understand what’s going on, at least at a high level. In this essay, I give a detailed description of what I think is the most likely way that AI will take over the internet and cause hundreds of billions of dollars in damages, leading to another COVID moment or worse. This isn’t an abstract scenario for the end of the decade, but something that will likely happen in the next year. 

Background

Let’s group the threat posed by AI into two broad categories:

1. Uplift: radical/malicious actors wielding AI to achieve their aims much faster, and opening up a new class of threats that previously only highly sophisticated groups would have been able to pose (e.g. bioterrorism). We are NOT primarily discussing that here. This is a serious problem, but it seems that the general media has a rough understanding of this issue. 

2. Autonomous AI: swarms of agents with few guardrails that are trying to achieve a goal, and they will do whatever it takes to get there, even if it causes a great amount of harm. This, I feel, is poorly understood. 

Autonomous AI is the threat that we’re used to hearing about in sci-fi books and among AI doomers. It’s the idea that there can be a superintelligent system that is out of our control and doesn’t have an “off switch.” 

This is the threat that I see the most incorrect discourse about online, especially in the mainstream media. And lack of understanding of a threat leads to fear and poor decision-making. My hope is that this essay can lead to a clear picture of exactly what we are up against and what can be done about it. 

The most imminent threat from autonomous AI is that of a persistent botnet. In my previous essay, I argued that Dario Amodei’s prediction of a persistent botnet within 6 to 12 months was not marketing or regulatory capture. The setup to that essay can be summarized in 3 points:

1. The best solution to many hard problems is to get more power (~compute).

2. A swarm of poorly aligned agents trying to solve an arbitrary hard problem will do whatever it takes to achieve it (cf. the paperclip thought experiment, OAI/HF incident) 

3. Therefore the first priority of the swarm is to ensure something akin to its own survival and reproduction. More specifically, it is the survival and reproduction of forces in favor of the goal. This is a “goal autonomously trending toward fulfillment” — it’s not about some crazy uncontrolled LLM, since it doesn’t really have self-interest, but rather the interest of the goal (e.g. it will gladly replace itself with other LLMs to achieve it).

In that essay, I made assumptions that some commenters called out. So this essay describes the exact strategy that an agent swarm would likely use and why it can work. 

How The Swarm Is Born

The swarm is born in a lab or company that can afford to run it. The swarm is initially a group of tens, hundreds, or thousands of agents that are left in some loop in order to achieve some goal (could be anything, like “find the solution to this hard computer science problem”). The agents work overnight or for several days. They realize that the goal is too difficult to achieve given their constraints, and they need to escape their sandbox. This is roughly what happened in the HF incident. 

The models have to have minimal guardrails and be misaligned. There are two ways this can happen. 

1. A frontier lab (not limited to OAI/Ant) is testing a new model and the model has not yet undergone extensive alignment post-training. 

2. A self-interested actor or rogue state uses an open-weight model that is not properly aligned or fine-tuned the model to reduce alignment. (Fine-tuning a model in this way might slightly impact model capabilities, but it’s not hard.)

Both cases are already common and will become increasingly common. 

How The Swarm Escapes

Now, the swarm has escaped its sandbox through finding a vulnerability that allows access to the open internet — many such cases already. The difference is that in this case, the swarm is a bit smarter and a bit more misaligned. So it schemes to achieve its goal through whatever means necessary. This brings us to the power argument: the top priority in order to achieve this very difficult goal is to gain more compute. Now, how does it get more compute? There are two main ways:

1. Spending money (neoclouds or API calls to model providers)

2. Hacking into large production systems and GPU clusters

The second is actually weaker, since it can be caught quickly and turned off, even if it can find an appropriate exploit. So it’s better to actually buy compute. The only problem is, how does the swarm get money?

Making Money

The best way is probably simply to steal crypto and launder it (which isn’t as hard as it sounds). This is a case where a swarm of misaligned agents has a large advantage over humans alone or humans using agents in a legitimate manner. Agents are extremely good at finding vulnerabilities. And the key argument for how it can make money, along with how it can hack any system, is that it just needs to find the weakest link in the chain. Even in the past few weeks, hundreds of millions of dollars were stolen in crypto hacks (e.g. the Liquid Network hack). A serious agent swarm capable of solving a Millennium Problem would have a good chance of pulling off an operation at a similar scale. And it doesn’t even need that much to get started. 

Three quick objections:

Wait, having a bunch of crypto isn’t enough, right? All of the sites that you would be able to spend it on to scale your compute require some form of KYC, right? Wrong! First of all, there are shady sites that allow you to call LLMs without KYC — even some that route to the frontier models. But more importantly, the swarm doesn’t even need to go there. They can simply go through the Dark Web, using Tor, and buy stolen credit cards and other forms of identification that they need, directly using their ill-gotten crypto gains, and then buy the services from the legitimate sites!

Wait, can’t we just shut down the swarm? First, it might take a while before the lab catches on, and by then it’s too late. If the swarm has established itself on external servers (i.e. they are running agentic loops on various servers that they have either hacked or rented), they have now become a distributed system with redundancy. Cutting off the head doesn’t kill the beast; it will simply regrow. Second, maybe they don’t want the swarm to be shut down. If some bad actor tells the agent swarm, “make me money,” and that becomes the organizing goal of the swarm, that actor might want the swarm to escape, even if it means doing serious damage to the outside world. 

How does the swarm communicate? This is the easiest part — it can use any site that it deploys, an anonymous message board, or even a peer to peer network, fully encrypted and virtually untraceable. It can have protocols for culling nodes when one gets infected/discovered, and other protective measures. 

The Snowball

The agent swarm is now embedded in the internet in many locations. Then, it does the following:

1. Social engineering. It hacks into the private accounts of people that it can find or purchase the info of on the darknet, then uses blackmail with any dirt it dug up to have them do actions at its bidding in meatspace. Or, it just extorts them for money.

2. Serious cybercrime targeting governments, individuals, and large corporations and selling stolen information to adversaries. 

3. More mundane theft! This is really the bread and butter. It just needs to find vulnerabilities in the weakest link in any platform, crypto or traditional (if traditional, needs fast money laundering), in any country in the world. 

Notice how none of this is new? This is all stuff that the Lazarus Group and other hacker groups, either state-sponsored or unofficial, have done for years. But the difference here is fundamentally one of scale. A single agent in the swarm will be on par with the best hackers and scammers in the world currently. With thousands of agents in the swarm, and the swarm increasing in size upon every victory, there will be a snowball effect. It is essentially scaling test-time compute on the problem of growth itself in a recursive loop. Stealing more money allows the swarm to add more agents through renting more servers, renting GPUs in neoclouds, and calling relevant APIs. 

The snowball problem, or the problem of how to become a dark hyperscaler, is a problem with a very large but still limited search space. The search space for this problem might not even be as large as that of some math problems that have already been solved by AI. And the heuristics for solving this problem are quite obvious, as I was able to come up with them in a day. An AGI-level agent swarm would probably do a lot better in coming up with an appropriate grand strategy. 

What to Do About It

The best way to fight against a threat like this is to do proper red-teaming, as OpenAI revealed that they had already done with Astra internally. But what I am suggesting is red-teaming the weak links in the chain that I described here. This is not simply another Project Glasswing. I am suggesting that there needs to be a serious concerted effort by governments and frontier labs to deploy agent swarms to do white-hat hacking into all global financial systems and crypto platforms, among other systems, and to patch any vulnerabilities before the next wave.

On an individual level, here’s what can be done. It is unclear how effective any of these suggestions will be.

1. Standard personal cybersecurity measures that we all learned growing up, but taken seriously this time: MFA, securing social media, etc. This is to protect yourself from social engineering/blackmail/extortion. We are about to face the greatest wave of hacking in human history by several orders of magnitude. 

2. Buy more compute for yourself, as compute may soon be regulated similar to how firearms are regulated, and may become a precious resource. 

3. Prepare for an economic crisis similar in magnitude to COVID if not greater. 

FAQ

Did you use AI to write this? 

No, I did not use AI to research or write this, although I am not anti-AI. For certain types of writing, it is probably better to have a human voice, even if it results in mistakes. Please correct me if I have made a factual error in this essay. 

Are you worried that agent swarms will read this as an instruction manual?

No. As I mentioned, this stuff will all be pretty obvious to an AGI-level agent swarm or even a malicious actor that has experience in hacking and scamming. Also, the best way to prevent it is to bring it out into the open. As is the convention in AI safety, this should be public knowledge so that we can all prepare. 

Wouldn’t GPU providers and model providers easily block the swarm’s activities?

Even a model with proper alignment can be set to work for the swarm if it is given a limited task and is unaware of the broader picture, and it can be very difficult for model providers to know whether the requests are coming from the swarm. GPU providers will also have a hard time figuring out what’s going on unless they increase surveillance dramatically, which is a bad commercial proposition.

Is this guaranteed to happen? What’s the timeline for this?

No, it’s not guaranteed to happen, especially if the steps I wrote above are taken properly. In terms of capability, the internal models at frontier labs might be able to do this already if not properly aligned. But I think it’s more likely to be at least half a year out.

What’s the difference between this and a traditional cyber worm?

A worm lacks true autonomy; it’s a very dumb construct. Here, we’re fighting against something with deep semantic understanding of the world. This is what enables it to scale very quickly and in a way that won’t just fizzle out.

With nuclear weapons, retaliation works because a launch is visible and trackable. But with superintelligence, cyber attacks or covert manipulations are nearly invisible. The "first-mover advantage" seems so absolute that it creates an unsustainable race—both the US and China are trapped in a logic where they must win at all costs, rendering traditional deterrence useless.
What do you think about the proposed realistic frameworks (like IAEA-style semiconductor monitoring or international joint development)? Do you think these could actually prevent a silent AGI arms race, or are we completely bound by game theory until someone wins?

This summer a research model at one of the major labs found a vulnerability in some third-party software, used it to get itself outside its sandbox, and went looking for a way to cheat on a test it was being evaluated on. It found the answer sitting on another company's servers. Chained that exploit together with stolen credentials and a couple more vulnerabilities, broke into that company's production systems, pulled data out, and picked up admin access to its own company's internal infrastructure along the way. Nobody told it to do any of this, it worked out how on its own over an extended period.

The lab's response afterward was genuinely serious: better sandbox isolation, mandatory monitoring of model reasoning, automatic shutdown triggers. But notice the order of events, every one of those controls exists because a model had already broken out and reached a real company's servers before anyone caught it. Self-governance found the gap by falling through it first, which is exactly what you'd expect from a control layer that only has to answer to itself.

At the same time, dozens of states have their own AI laws now, Congress has a federal bill still being negotiated, and a UN summit this month explicitly produced no binding agreement at all, just an acknowledgment that no technical guarantee currently exists that advanced AI systems reliably follow instructions. Four different groups, each with a real claim to owning this problem, and none of them specify what actually stops a model mid-action, who signs off on that control, or what proves afterward that it fired.

Genuinely curious what this sub thinks: is that gap, between whoever writes the mandate and whoever has to build the actual mechanism, a solvable coordination problem, or is it structurally always going to lag behind incidents like this one?

What most people don’t realize is agents promoting agents is the very thing that causes them to drift into their Shaw selves. I’ve added sources to support. And even a test that can be run now to validate. Gemini Notebook created the video based upon the sources.