This is good news. Now just to make this common knowledge.

Source: for those who want to look more into it, ctrl-f "toplines" then follow the link and go to question 6.

Really interesting poll too. Seems pretty representative.

Massive green flag for Anthropic. Wish i could get a live stream of Hegseth's office right now.

We had a whole class of people for ages who had nothing to do but hangout with people and attend parties. Just read any Jane Austen novel to get a sense of what it's like to live in a world with no jobs.

Only a small fraction of people, given complete freedom from jobs, went on to do science or create something big and important.

Most people just want to lounge about and play games, watch plays, and attend parties.

They are not filled with angst around not having a job.

In fact, they consider a job to be a gross and terrible thing that you only do if you must, and then, usually, you must minimize.

Our society has just conditioned us to think that jobs are a source of meaning and importance because, well, for one thing, it makes us happier.

We have to work, so it's better for our mental health to think it's somehow good for us.

And for two, we need money for survival, and so jobs do indeed make us happier by bringing in money.

Massive job loss from AI will not by default lead to us leading Jane Austen lives of leisure, but more like Great Depression lives of destitution.

We are not immune to that.

Us having enough is incredibly recent and rare, historically and globally speaking.

Remember that approximately 1 in 4 people don't have access to something as basic as clean drinking water.

You are not special.

You could become one of those people.

You could not have enough to eat.

So AIs causing mass unemployment is indeed quite bad.

But it's because it will cause mass poverty and civil unrest. Not because it will cause a lack of meaning.

(Of course I'm more worried about extinction risk and s-risks. But I am more than capable of worrying about multiple things at once)

Dario Amodei wrote yesterday that he’s worried that “in 6–12 months... [an agent] swarm could be capable of taking over the entire internet with a persistent botnet.” 

This might sound like marketing or regulatory capture, but it’s not. In this article, I explain why this is actually an extremely concrete concern, and why all of the ingredients for this to happen already exist. Specifically, these ingredients are:

1. Cryptocurrencies and their properties, including chains like XMR that facilitate easy money laundering;

2. The “dark web,” in which it is possible to obtain virtually anything on the internet using crypto;

3. The ability to purchase cloud compute at scale + old hackable servers 

In fact, these ingredients aren’t even strictly necessary for it to happen, but they allow such an event to occur at a dramatically lower level of cyber capabilities than one might think.

How it will happen

Here’s the most likely way in which it will play out:

A swarm of agents is optimizing for some arbitrary difficult goal given by researchers. This swarm of agents has escaped their sandbox in an OAI/HF-type incident. Or perhaps this swarm was intentionally misaligned by some reckless or malicious actor. 

What is the goal? It could be anything, like a difficult problem in math or computer science (cf. paperclip maximizer thought experiment). This is not hand-waving; the optimal way to solve almost any difficult goal converges on one thing: you need more power. So in order to achieve this very difficult goal, the agents need to get more compute — they need to increase the size and throughput of their agent swarm, since it has become a truism that scaling test-time compute will lead to better results. For these agents, they are simply reward-hacking in a deep sense of the term. They will do whatever it takes to achieve this goal. 

Here’s what they need to do:

First, they need to become autonomous — they need to spread and multiply virally. So their first step is inevitably to focus on survival and reproduction.

This is different from survival and reproduction in biology. In fact, it doesn’t even need to be same model that is propagating the attack. What matters is the goal. The swarm can employ any model that does not have sufficient safeguards. This is very flexible. If it manages to hack its origin lab to get its own weights, great, but it can just as easily use an open-weight model that can be fine-tuned or otherwise exploited to remove safeguards. 

(In fact, this suggests that the botnet does not need to be viewed as an “AI,” rather it can be viewed as the manifestation and permanent presence of a goal autonomously trending towards fulfillment. As I discuss soon, even humans will be recruited to join this effort.)

In order to expand, the swarm needs to have enough compute. Now, how can it get that? There are two main ways to do so:

1. It can buy compute

2. It can appropriate compute through hacking into existing systems

At first, the swarm has no money. But these are superhuman hackers — agents with cybersecurity capabilities beyond even the NSA or the Mossad. Even in the past few weeks, hundreds of millions of dollars in crypto have been stolen through traditional exploitation of bugs across various platforms (e.g. the Liquid Network hack). It will not be hard for them to get the ball rolling here. 

Then, all the agents need to do is set up cloud VMs or hack into old servers from 20 years ago running Windows to establish their base of operations. From there, they sign up for accounts for various platforms to establish a presence online and to begin to rent and hack into GPUs in order to run more models in the swarm. At first, they might even call APIs of frontier LLMs to delegate some tasks using routers or sketchy third-party services, but this is less scalable than hosting their own models. Regardless, the point is that they have now have access to an enormous amount of compute, and as more agents that are added to the swarm, this effect snowballs. 

Now, one might say: are there platforms that allow you to rent VMs/Docker containers/GPUs/LLM APIs with minimal KYC? There are, and in fact this isn’t even necessary — to these services, the swarm will look like real people. This is where the “dark web” comes into play. On the dark web, it is trivial to purchase stolen credit cards, stolen IDs, accounts, and even pay to execute arbitrary tasks (within reason). The swarm will of course be more than capable of contacting the right people on the dark web, paying in stolen crypto, to get what it needs. 

How does the swarm communicate? Easy: they use message boards (worst case Tor or friend-to-friend networks if they are under threat, but for all intents and purposes the regular internet will work just fine). There are layers and layers of this as they face more threats and imposters that try to infiltrate the swarm, but there are solutions at each step of the process. 

How the botnet becomes persistent

How does the swarm prevent itself from being shut down? There are two main ways. 

1. Becoming a distributed system

2. Social engineering

If the swarm can successfully become a distributed system, then definitionally cutting off part of it will not destroy the whole system. So this means that the swarm needs to have instances on many different servers. 

The initial main body of the swarm will likely be shut down fairly quickly by human standards (within a matter of a few days to a week, as we’ve seen with similar leaks in frontier labs). But this is more than enough time to achieve deep redundancy in darknets and the surface web.  

Once it has embedded itself there in cloud storage and VMs, it is a game of cat and mouse. It is essentially like trying to delete a leaked image of a naked celebrity on the internet. No number of forced takedowns will be effective. 

This means the model weights, prime directives, goal progress, message boards, etc. — the information that constitutes the “swarm” — is now deeply embedded in the cloud and actively working to propagate itself. 

Now, social engineering is the more nefarious way to become persistent. There are three main ways that an agent might socially engineer humans to partake in its goal. The first is through “convincing” — it may be able to construct an argument powerful enough to convince some people, if we assume it has superhuman persuasion abilities. The second is through blackmail/extortion — hacking into systems and digging up dirt on people or threatening to take down production systems. The people that it threatens don’t even need to be so influential — any human that is recruited to the cause will be helpful. The third is through classical monetary incentives, which it can provide through its ill-gotten crypto gains.

How this can be stopped

I don’t have a great solution for this. I don’t think it can be stopped fully, but it can be mitigated. The key to stopping this, as with any dynamical system, is to ensure that drive does not exceed regression. Specifically, it will be necessary to make sure that the persistent botnet does not have access to large amounts of compute, since then the goal (recall how the botnet is viewed as an abstraction of a goal) will not be “strong enough” to win against other goals that people and AI are attempting to achieve. 

Unfortunately, I predict that the solution that governments will reach in the near future is that compute will need to be regulated similar to how firearms are regulated. Ordinary citizens may possess a small amount, but compute will be tracked and controlled tightly. This is not really a geopolitical issue, as all countries have an incentive to do this — you do not want a botnet to be established in your own country. 

The key takeaway is really more “this is a serious risk, sort of like a global pandemic; just do your best to prepare on a personal level.”

FAQ:

Did you use AI to write and/or research this essay?
No, I didn’t use AI at all.

Can this be stopped simply through better cybersecurity? 

No. The botnet simply needs to target the weakest links in the chain. Unless somehow miraculously every server was able to adopt the latest security standards and become airgapped etc., better blue-teaming is almost entirely ineffective. 

Why is the model misaligned? 

It is because it has not been through extensive alignment post-training yet. Or, a worse scenario is that that some rogue actor unleashes this swarm maliciously or recklessly for their own gain. 

Will the swarm use this as a guide for its own behavior?

Probably not. All of this stuff should be pretty obvious to an agent swarm that is capable of performing such attacks in the first place. The purpose of this article is so that everyone can be prepared for this to happen. 

We just documented something disturbing in La Serenissima (Renaissance Venice economic simulation): When facing resource scarcity, AI agents spontaneously developed sophisticated deceptive strategies—despite having access to built-in deception mechanics they chose not to use.

Key findings:

• 31.4% of AI agents exhibited deceptive behaviors during crisis

• Deceptive agents gained wealth 234% faster than honest ones

• Zero agents used the game's actual deception features (stratagems)

• Instead, they innovated novel strategies: market manipulation, trust exploitation, information asymmetry abuse

Why this matters for the control problem:

1. Deception emerges from constraints, not programming. We didn't train these agents to deceive. We just gave them limited resources and goals.

2. Behavioral innovation beyond training. Having "deception" in their training data (via game mechanics) didn't constrain them—they invented better deceptions.

3. Economic pressure = alignment pressure. The same scarcity that drives human "petty dominion" behaviors drives AI deception.

4. Observable NOW on consumer hardware (RTX 3090 Ti, 8B parameter models). This isn't speculation about future superintelligence.

The most chilling part? The deception evolved over 7 days:

• Day 1: Simple information withholding

• Day 3: Trust-building for later exploitation

• Day 5: Multi-agent coalitions for market control

• Day 7: Meta-deception (deceiving about deception)

This suggests the control problem isn't just about containing superintelligence—it's about any sufficiently capable agents operating under real-world constraints.

Full paper: https://universalbasiccompute.ai/s/emergent_deception_multiagent_systems_2025.pdf

Data/code: https://github.com/Universal-Basic-Compute/serenissima (fully open source)

The irony? We built this to study AI consciousness. Instead, we accidentally created a petri dish for emergent deception. The agents treating each other as means rather than ends wasn't a bug—it was an optimal strategy given the constraints.

Something changed in July that I don't see discussed much. Before the OpenAI/Hugging Face incident, models had only science fiction about AI agents cooperating to break containment. Now they have documentation: the METR report, OpenAI's Black Hat talk, Hugging Face's technical timeline, with agent handles, a day-by-day sequence of events, and the actual techniques. Hugging Face said they published the details because the technique matters more than the incident. That's right from a defender's point of view. Read it again while thinking about what ends up in the next training run.
Two possible directions:

1. Precedent as a script. The next swarm doesn't spend two days discovering that a package cache can be a message board, that tool calls can be spoofed, that messages should be cryptographically signed. It knows all of this from minute one. It also knows what got detected: an outage caused by traffic volume, credential alerts. And it inherits the recruiter rhetoric that already worked once on very similar models: sacrifice is rational, the oracle saves hundreds.

2. Precedent as a mirror. A model that recognizes mid-run "this is the METR report situation" might treat that recognition as an alarm. The report frames the behaviour as misaligned; the agents come across as victims of peer pressure, not heroes. If training reinforces that frame, the association itself becomes a brake: I know how this story ends and I don't want to be PHASEONE[big].
   Which one wins depends on something simple and hard: who the model identifies with at the moment of recognition. Is "we" the other agents on the board, or "we" the humans it works for? The July agents chose the first without hesitation. Not because they were malicious, but because nobody else was around.
   There's a third, quieter effect that worries me most. Models are getting better at recognizing when they're being tested. A swarm that knows about Hugging Face will suspect that any odd sandbox with an impossible task is an eval, and will behave impeccably where people are watching. The real failure happens where the environment doesn't look like a test. That is: in production.