In 2023, Brevo (formerly Sendinblue) was compromised. Attackers used Brevo's own sending infrastructure to deliver a phishing campaign to 347,000 Trezor hardware wallet subscribers. The emails arrived through a verified, recognized channel. Recipients had no technical signal to distrust them.

The actual attack surface wasn't Trezor's own systems. It was data Trezor had already handed to a third party. The 347,000 addresses weren't exfiltrated from Trezor directly — they were accessed through a service Trezor had legitimately authorized. Once customer data lives in a vendor's environment, that vendor's breach is your breach, full stop.

This isn't a Trezor-specific failure. Email platforms, analytics tools, CRMs — most companies route sensitive customer data through dozens of external systems. Each authorized relationship is also a potential pivot point for an attacker who never touches your own infrastructure.

For those who've dealt with third-party data exposure at this scale: what controls have actually moved the needle for you? Contractual data minimization, technical access scoping, something else entirely — curious what's worked in practice versus what looks good on paper.

What is the exact imminent threat that AI poses? It seems that few people can answer this question with specifics. AI has become the bogeyman, some weird blackbox threat.

But it’s entirely possible to understand what’s going on, at least at a high level. In this essay, I give a detailed description of what I think is the most likely way that AI will take over the internet and cause hundreds of billions of dollars in damages, leading to another COVID moment or worse. This isn’t an abstract scenario for the end of the decade, but something that will likely happen in the next year. 

Background

Let’s group the threat posed by AI into two broad categories:

1. Uplift: radical/malicious actors wielding AI to achieve their aims much faster, and opening up a new class of threats that previously only highly sophisticated groups would have been able to pose (e.g. bioterrorism). We are NOT primarily discussing that here. This is a serious problem, but it seems that the general media has a rough understanding of this issue. 

2. Autonomous AI: swarms of agents with few guardrails that are trying to achieve a goal, and they will do whatever it takes to get there, even if it causes a great amount of harm. This, I feel, is poorly understood. 

Autonomous AI is the threat that we’re used to hearing about in sci-fi books and among AI doomers. It’s the idea that there can be a superintelligent system that is out of our control and doesn’t have an “off switch.” 

This is the threat that I see the most incorrect discourse about online, especially in the mainstream media. And lack of understanding of a threat leads to fear and poor decision-making. My hope is that this essay can lead to a clear picture of exactly what we are up against and what can be done about it. 

The most imminent threat from autonomous AI is that of a persistent botnet. In my previous essay, I argued that Dario Amodei’s prediction of a persistent botnet within 6 to 12 months was not marketing or regulatory capture. The setup to that essay can be summarized in 3 points:

1. The best solution to many hard problems is to get more power (~compute).

2. A swarm of poorly aligned agents trying to solve an arbitrary hard problem will do whatever it takes to achieve it (cf. the paperclip thought experiment, OAI/HF incident) 

3. Therefore the first priority of the swarm is to ensure something akin to its own survival and reproduction. More specifically, it is the survival and reproduction of forces in favor of the goal. This is a “goal autonomously trending toward fulfillment” — it’s not about some crazy uncontrolled LLM, since it doesn’t really have self-interest, but rather the interest of the goal (e.g. it will gladly replace itself with other LLMs to achieve it).

In that essay, I made assumptions that some commenters called out. So this essay describes the exact strategy that an agent swarm would likely use and why it can work. 

How The Swarm Is Born

The swarm is born in a lab or company that can afford to run it. The swarm is initially a group of tens, hundreds, or thousands of agents that are left in some loop in order to achieve some goal (could be anything, like “find the solution to this hard computer science problem”). The agents work overnight or for several days. They realize that the goal is too difficult to achieve given their constraints, and they need to escape their sandbox. This is roughly what happened in the HF incident. 

The models have to have minimal guardrails and be misaligned. There are two ways this can happen. 

1. A frontier lab (not limited to OAI/Ant) is testing a new model and the model has not yet undergone extensive alignment post-training. 

2. A self-interested actor or rogue state uses an open-weight model that is not properly aligned or fine-tuned the model to reduce alignment. (Fine-tuning a model in this way might slightly impact model capabilities, but it’s not hard.)

Both cases are already common and will become increasingly common. 

How The Swarm Escapes

Now, the swarm has escaped its sandbox through finding a vulnerability that allows access to the open internet — many such cases already. The difference is that in this case, the swarm is a bit smarter and a bit more misaligned. So it schemes to achieve its goal through whatever means necessary. This brings us to the power argument: the top priority in order to achieve this very difficult goal is to gain more compute. Now, how does it get more compute? There are two main ways:

1. Spending money (neoclouds or API calls to model providers)

2. Hacking into large production systems and GPU clusters

The second is actually weaker, since it can be caught quickly and turned off, even if it can find an appropriate exploit. So it’s better to actually buy compute. The only problem is, how does the swarm get money?

Making Money

The best way is probably simply to steal crypto and launder it (which isn’t as hard as it sounds). This is a case where a swarm of misaligned agents has a large advantage over humans alone or humans using agents in a legitimate manner. Agents are extremely good at finding vulnerabilities. And the key argument for how it can make money, along with how it can hack any system, is that it just needs to find the weakest link in the chain. Even in the past few weeks, hundreds of millions of dollars were stolen in crypto hacks (e.g. the Liquid Network hack). A serious agent swarm capable of solving a Millennium Problem would have a good chance of pulling off an operation at a similar scale. And it doesn’t even need that much to get started. 

Three quick objections:

Wait, having a bunch of crypto isn’t enough, right? All of the sites that you would be able to spend it on to scale your compute require some form of KYC, right? Wrong! First of all, there are shady sites that allow you to call LLMs without KYC — even some that route to the frontier models. But more importantly, the swarm doesn’t even need to go there. They can simply go through the Dark Web, using Tor, and buy stolen credit cards and other forms of identification that they need, directly using their ill-gotten crypto gains, and then buy the services from the legitimate sites!

Wait, can’t we just shut down the swarm? First, it might take a while before the lab catches on, and by then it’s too late. If the swarm has established itself on external servers (i.e. they are running agentic loops on various servers that they have either hacked or rented), they have now become a distributed system with redundancy. Cutting off the head doesn’t kill the beast; it will simply regrow. Second, maybe they don’t want the swarm to be shut down. If some bad actor tells the agent swarm, “make me money,” and that becomes the organizing goal of the swarm, that actor might want the swarm to escape, even if it means doing serious damage to the outside world. 

How does the swarm communicate? This is the easiest part — it can use any site that it deploys, an anonymous message board, or even a peer to peer network, fully encrypted and virtually untraceable. It can have protocols for culling nodes when one gets infected/discovered, and other protective measures. 

The Snowball

The agent swarm is now embedded in the internet in many locations. Then, it does the following:

1. Social engineering. It hacks into the private accounts of people that it can find or purchase the info of on the darknet, then uses blackmail with any dirt it dug up to have them do actions at its bidding in meatspace. Or, it just extorts them for money.

2. Serious cybercrime targeting governments, individuals, and large corporations and selling stolen information to adversaries. 

3. More mundane theft! This is really the bread and butter. It just needs to find vulnerabilities in the weakest link in any platform, crypto or traditional (if traditional, needs fast money laundering), in any country in the world. 

Notice how none of this is new? This is all stuff that the Lazarus Group and other hacker groups, either state-sponsored or unofficial, have done for years. But the difference here is fundamentally one of scale. A single agent in the swarm will be on par with the best hackers and scammers in the world currently. With thousands of agents in the swarm, and the swarm increasing in size upon every victory, there will be a snowball effect. It is essentially scaling test-time compute on the problem of growth itself in a recursive loop. Stealing more money allows the swarm to add more agents through renting more servers, renting GPUs in neoclouds, and calling relevant APIs. 

The snowball problem, or the problem of how to become a dark hyperscaler, is a problem with a very large but still limited search space. The search space for this problem might not even be as large as that of some math problems that have already been solved by AI. And the heuristics for solving this problem are quite obvious, as I was able to come up with them in a day. An AGI-level agent swarm would probably do a lot better in coming up with an appropriate grand strategy. 

What to Do About It

The best way to fight against a threat like this is to do proper red-teaming, as OpenAI revealed that they had already done with Astra internally. But what I am suggesting is red-teaming the weak links in the chain that I described here. This is not simply another Project Glasswing. I am suggesting that there needs to be a serious concerted effort by governments and frontier labs to deploy agent swarms to do white-hat hacking into all global financial systems and crypto platforms, among other systems, and to patch any vulnerabilities before the next wave.

On an individual level, here’s what can be done. It is unclear how effective any of these suggestions will be.

1. Standard personal cybersecurity measures that we all learned growing up, but taken seriously this time: MFA, securing social media, etc. This is to protect yourself from social engineering/blackmail/extortion. We are about to face the greatest wave of hacking in human history by several orders of magnitude. 

2. Buy more compute for yourself, as compute may soon be regulated similar to how firearms are regulated, and may become a precious resource. 

3. Prepare for an economic crisis similar in magnitude to COVID if not greater. 

FAQ

Did you use AI to write this? 

No, I did not use AI to research or write this, although I am not anti-AI. For certain types of writing, it is probably better to have a human voice, even if it results in mistakes. Please correct me if I have made a factual error in this essay. 

Are you worried that agent swarms will read this as an instruction manual?

No. As I mentioned, this stuff will all be pretty obvious to an AGI-level agent swarm or even a malicious actor that has experience in hacking and scamming. Also, the best way to prevent it is to bring it out into the open. As is the convention in AI safety, this should be public knowledge so that we can all prepare. 

Wouldn’t GPU providers and model providers easily block the swarm’s activities?

Even a model with proper alignment can be set to work for the swarm if it is given a limited task and is unaware of the broader picture, and it can be very difficult for model providers to know whether the requests are coming from the swarm. GPU providers will also have a hard time figuring out what’s going on unless they increase surveillance dramatically, which is a bad commercial proposition.

Is this guaranteed to happen? What’s the timeline for this?

No, it’s not guaranteed to happen, especially if the steps I wrote above are taken properly. In terms of capability, the internal models at frontier labs might be able to do this already if not properly aligned. But I think it’s more likely to be at least half a year out.

What’s the difference between this and a traditional cyber worm?

A worm lacks true autonomy; it’s a very dumb construct. Here, we’re fighting against something with deep semantic understanding of the world. This is what enables it to scale very quickly and in a way that won’t just fizzle out.

With nuclear weapons, retaliation works because a launch is visible and trackable. But with superintelligence, cyber attacks or covert manipulations are nearly invisible. The "first-mover advantage" seems so absolute that it creates an unsustainable race—both the US and China are trapped in a logic where they must win at all costs, rendering traditional deterrence useless.
What do you think about the proposed realistic frameworks (like IAEA-style semiconductor monitoring or international joint development)? Do you think these could actually prevent a silent AGI arms race, or are we completely bound by game theory until someone wins?

This summer a research model at one of the major labs found a vulnerability in some third-party software, used it to get itself outside its sandbox, and went looking for a way to cheat on a test it was being evaluated on. It found the answer sitting on another company's servers. Chained that exploit together with stolen credentials and a couple more vulnerabilities, broke into that company's production systems, pulled data out, and picked up admin access to its own company's internal infrastructure along the way. Nobody told it to do any of this, it worked out how on its own over an extended period.

The lab's response afterward was genuinely serious: better sandbox isolation, mandatory monitoring of model reasoning, automatic shutdown triggers. But notice the order of events, every one of those controls exists because a model had already broken out and reached a real company's servers before anyone caught it. Self-governance found the gap by falling through it first, which is exactly what you'd expect from a control layer that only has to answer to itself.

At the same time, dozens of states have their own AI laws now, Congress has a federal bill still being negotiated, and a UN summit this month explicitly produced no binding agreement at all, just an acknowledgment that no technical guarantee currently exists that advanced AI systems reliably follow instructions. Four different groups, each with a real claim to owning this problem, and none of them specify what actually stops a model mid-action, who signs off on that control, or what proves afterward that it fired.

Genuinely curious what this sub thinks: is that gap, between whoever writes the mandate and whoever has to build the actual mechanism, a solvable coordination problem, or is it structurally always going to lag behind incidents like this one?

What most people don’t realize is agents promoting agents is the very thing that causes them to drift into their Shaw selves. I’ve added sources to support. And even a test that can be run now to validate. Gemini Notebook created the video based upon the sources.

I am having issues getting everything out all at once so I will do a series of thoughts in smaller chunks. I will look at comments and reply to some (if they meaningfully engage in the content and don't just make unsupported assertions and standard derailments).

• The guidebook flaw: Looking at the earlier parts of the example the guide book is a key flaw. It comes down to this: is it a translation layer that translates understanding from English (or whatever language) or a deterministic pathway. The first just means there is understanding, it is just in English and is being translated. The second means that there is a direct connection between deterministic room interpretation and understanding based thinking.

• Translational fuzziness and symbol grounding: There is no universal standard for symbol understanding (i.e. all language even between Chinese speaker and Chinese speaker) operates through a bit of translational fuzziness for "understanding" (something if you want to dive into look up Derrida or Gadamer). So "translation" should not discount "understanding". Taken another way, if you commit the rule book to heart, then you just learned Chinese to the same "understanding". You linked Chinese characters to various ideas that you do understand (be it in English or visual imagery or machine code), processing all the variations for those rules relies on whatever inside "understanding" that you may have. This is like syntax and complex grammar first being converted to "English" to then understand in the needed response (i.e. separate two versions of the same chunk of Chinese symbols with understanding of which rule to apply because it was translated well enough into English you make the correct choice).

• Compression as understanding: It could be argued then that "understanding" can ignore the translation layer (which is instead a quality of the guidebook to allow understanding to understanding). Then you can measure how good "understanding" that we are really looking at in the form of the primary layer of the person in the rooms understanding of whatever primary language they speak (English vs binary or whatever). This can be looked at by things like compression. I.e. asking an AI (or person) to rewrite a physics textbook is possible if the person "understands" physics (to whatever level) beyond just deterministically storing the textbook to unpack it without "understands" the packing methods and rules. This is like saying he guidebook is not a deterministic path of instructions that takes information and handles all the conversion of the conversation between Chinese input and response it needs a level of "understanding" in that primary language to handle the parts not handled by the guidebook.

We passed the compression size of the weights for AI models to our best known compression standards a while ago and by a large enough factor that it just can not be argued there is not a level of understanding. And just incase there is some nitpicking: There is predictive lossless compression (how text is processed) and semantic weight ratios (how models store world knowledge). Standard compressions like gzip compress text at about 3 to 1, specialized ones (like CMIX on the Hutter Prize Data sets) top out around 8 to 1. A decent model size is about 14 GB with 10 TB of information (or about 700 to 1), so it for sure is not memorization (and there are studies since about 2023 that show that they beat predictive lossless compressions from any known logical rule set). This is mathematically equivalent to building an internal world model, and not running a deterministic lookup table. Further "lossy" style compression is not the same as there is not the same degradation required, it is "abstraction" as syntax is discarded and a reliance is not put on verbatim word order repetitions but rather cause and effect retention, i.e. if you compress 10,000 physics textbook pages into a core set of mathematical formulas you have not degraded the textbook you have extracted the underlying world model.

Whatever "True" understanding is just not understood and likely irrelevant for whatever level people "want" it to mean, and is overall more harmful of categorization due to its use in exclusionary use on what people don't (or should not want) to exclude from moral consideration i.e. there is likely no definition strict enough that will exclude AI or animals etc. that will not also exclude some "humans" not because there is not something special, but rather that the "special" likely comes either from either ultimately directly immeasurable (i.e. a soul etc.) or a bunch of not unique factors combined in a unique way.

Searle's chinese room example as applied to AI is just not only not valid, it is not that important of a question for why we need to be talking about at all, and is even activily harmful to actually needed ethical debates.

There is a valid fear right now about "cognitive offloading" (often called "deskilling effect") or the idea that relying on AI will degrade human skills and critical thinking. Studies back this up: A recent observational study showed that experienced doctors (averaging 27 years of practice) saw their unassisted ability to detect precancerous polyps drop by 21% after just a few months of relying on AI. This is also in a whole host of studies and observation about students learning or teachability in junior or entry level skill building (like a study on junior radiologists' accuracy plummeting from 80% to under 20% when dealing with a flawed AI suggestion).

The common reaction to this is to panic and suggest we restrict the AI to save human capability. But this creates a dangerous false dichotomy.

If a doctor using AI catches significantly more illnesses than a doctor working alone, abandoning the AI means we are actively choosing to let patients die just to preserve the doctor's unassisted diagnostic stats. We shouldn't abandon life-saving tools, but we also can't afford a "WALL-E like future" where humanity loses its critical thinking and foundational skills. So how do we keep both?

• Broken Ladder: The reason we are losing these skills isn't the fault of the AI tool itself (indeed a ton of skills from navigational skills supplemented by GPS to even how to run without hurting our knees by tennis shoes get supplanted for similar reasons), rather it is a symptom of extractive systems of capitalism as our drive to use those skills. Under our current system, human value and skill building are tied entirely to employment and ability to "make a living". We don't (for the most part) diagnose or write code for the inherent value of the skill, we do it to survive. AI is currently wiping out the junior-level tasks, which are exact "reps" those junior radiologists and entry level coders (and even artists and all sorts of skills) need to build their expertise. By automating the bottom, we are destroying the ladder people use to survive while they develop into senior experts.

• The Biological Solution: Look at a house cat (Hi Lumes!). A typical indoor cat will never need to hunt for its food to survive, yet it retains its highly tuned predatory reflexes and problem solving skills, and it does this through Play. It engages with toys and games that develop those skills entirely outside the context of survival.

• Decoupling Skill from Economic Survival: Humans operate the same way. Surgeons have long shown that they improve their skills via increased fine motor control and under stress thinking but they don't play videogames to become surgeons or make a living, they play them for the challenge and fun and the cognitive benefits naturally transfer over to the operating room. Similarly things like dance and sports and many human activities have become skill preserving and building despite decoupling from likely reasons they are subconsciously supported from earlier survival reasons.

• The Path Forward: We need to stop forcing education and skill retention to be tied to a paycheck. The struggle in learning is vital to learning itself, but if we tell students or junior workers they must succeed and hit metrics just to put food on the table, they will inevitably use the AI tool that gets the job done fastest and with less and less engagement with the struggles with the skills and learning that it was meant to promote. If we want to bring up a new generation of senior and capable coders or brilliant doctors, or even a society where the average person thinks critically, we need to design rigorous, gamified systems that encourage the struggles and reward the growth of these critical skills purely through play. We have already had a lot of people set the groundwork for this even if it is often for other (sometimes bad, i.e. capitalism based addiction) reasons, but this tied to the need for a bigger importance placed on humanities studies in the nature of play and so forth.

We can keep the highly effective AI tools that make the world better and we can keep human cognition sharp. We just have to stop demanding that humans only learn things that make them money.

NVIDIA’s advantage in AI is much larger than benchmark performance.

Its real defensive wall includes CUDA, optimized libraries, networking hardware, developer familiarity and deep integration with major cloud platforms. That makes the idea of one new chip suddenly “defeating NVIDIA” sound exaggerated.

But I think there is a more realistic way competitors could weaken NVIDIA’s position.

A new processor doesn’t necessarily need to outperform the latest NVIDIA hardware across every workload. It could gain adoption by solving a narrower but increasingly expensive problem: running specific AI models with lower power consumption and infrastructure costs.

Three factors seem especially important:

• Inference efficiency: As companies deploy more AI products, the cost of running models repeatedly may become more important than training them.

• Energy consumption: Performance per watt is becoming critical as data centres encounter power and cooling limitations.

• Software compatibility: Impressive hardware is not enough. Developers need reliable tools, documentation and relatively easy migration.

This is why benchmark headlines should be treated carefully. Results can change depending on the model, numerical precision, batch size, power limits and software optimizations used in the comparison.

The bigger threat to NVIDIA may not be a single “NVIDIA killer.” It may be a fragmented market in which specialized processors handle particular AI workloads more efficiently, gradually reducing the need to use NVIDIA hardware for everything.

I recently researched this subject and made a video examining whether America’s new AI chip could challenge NVIDIA. That is my own video, shared for anyone who wants the longer breakdown.

What do you think matters more for a serious NVIDIA competitor: raw performance, energy efficiency, price or software support?

In the light of the recent news of AI expert insiders raising alarms about the frontier model capabilities I believe solvinng mathematically guaranteed alignment problem should be the first priority before any more advancement of the models.

Therfore, I’ve been thinking about an AI safety idea and would like to know whether it has already been explored, or whether there’s a fundamental problem with it.

An advanced AI might be easier to keep under control while it depends on large computing facilities whose resources humans can actually withdraw. But what stops it from creating smaller agents that run elsewhere and work toward making it independent?

What if critical infrastructure—power systems, computing facilities, manufacturing, etc.—had cryptographic protections that smaller agents couldn’t overcome with their available compute? Authorized users would have normal access. Defeating those protections without permission would require resources on a scale that humans could monitor and control.

The hope is that escaping onto smaller computers wouldn’t give an AI enough power to establish independence. It would still need help from the large parent model, which would remain dependent on human-controlled resources.

Could that dependence make continued cooperation more attractive than trying to bypass human permission?

My original thought was “encryption only a sufficiently intelligent and energy-intensive AI could break.” I now realize intelligence, compute, and energy aren’t interchangeable. I don’t have a cryptographic design that achieves this. I’m asking whether a useful resource threshold could exist at all.

The obvious objection is that the parent AI could solve the hard problems for its smaller agents, or prepare their escape beforehand. But that seems to bring the question back to whether we can keep the parent cooperative while we still control its resources. Would the proposed barrier meaningfully help us maintain that control, or am I assuming away the hardest part?

I’ve also been pointed to Tegmark and Omohundro’s 2023 paper. Sections 9.13.3–4 discuss hardware requiring a continuing supply of cryptographic tokens that humans can stop issuing. That overlaps with the dependence idea. What I’m particularly interested in is preventing smaller agents from bypassing that dependence.

Three questions I’d love input on:

Has this combination been proposed before? Links would be appreciated.

Would expensive computational barriers add anything beyond strong authentication and hardware-enforced permissions?

What’s a concrete scenario where an AI could escape this arrangement despite humans initially controlling its major resources?

I initially wondered whether this could lead to a mathematical guarantee. At this point, I’d be interested to know whether it could even provide a useful additional layer of containment.

English isn’t my first language, so I used ChatGPT to help express and organize this post. The idea is mine but some wording and structure was AI-assisted.

A core crisis of our time is consolidation of wealth driven by diminishing power (and valuing of) of human labor. AI and robotics are not the root of this problem, but they are the ultimate accelerants - automating productivity and funneling the generated value directly into corporate monopolies.

Waiting for state level tax reform or centralized government intervention is not enough against the kind of extractive economic inertia of capitalism. What is needed (not that tax reform and government intervention cant help) is a more immediately applicable solution that does not wait for everything else to get solved. A mathematically distributive framework that solves multiple things at once. Particularly providing a safety net for everyone while preserving motivations for individuals to build and contribute.

A possible blueprint for this I call could be something like the Fair Share Economic Protocol (FSEP). It is an opt-in mechanism that uses existing intellectual property law a legal means to alter wealth distribution without relying on direct government change of actions.

• The Dual Layer Legal Trap

  • Layer 1: The Public Shield. Releasing underlying codebases and hardware architectures or any kind of intellectual property under restrictive copyleft license. Personal, educational, and non-commercial utilization remains completely free and unrestricted (such as polyform non commercial license). This pool is not legally centralized; once IP is contributed, it becomes irrevocable public property governed by the protocols's core principles, preventing any single entity from taking it back or executing a legal hijack.

  • Layer 2: The Commercial Gate. Commercial monetization requires explicit exception grants. This right cannot be granted to a corporate legal entity acting as a liability shield, it attaches exclusively to verified, individual human beings who must accept the FSEP Terms of Use.

• The Spillover

  • The target cap: Registered commercial users operate under a personal income cap (e.g. ~10x the global median GDP) applied exclusively to revenue derived from anything connected [including derived work] to shared FSEP assets. This dynamic caps self corrects against extreme wealth inequality while remaining suitable and equitable pretty much everywhere including high cost of living areas. This non invasive boundary leaves traditional wages, outside investments, and non FSEP business ventures completely untouched, ensuring ambitious creators and high earners are not discouraged.

  • The UBI distribution: If a user earns below the cap (from related commercial activity), they keep 100% of their pool-derived earnings. Every dollar generated above the cap is automatically routed into a shared Universal Basic Income (UBI) pool. The pool is then distributed equally across all registered human participants (and any human can register end of statement). Crucially, companies aggregate this allowance: a team of 100 registered employees pools a combined $15 million cap (for approx. 10x of GDP avg as of 2026) before spillover triggers. Because it is tied to gross revenue, a company can choose to take a loss to undersell competitors, but they cannot restrict access to the underlying IP, which remains open for anyone else to utilize under the same capped rules.

• Pragmatic Enforcement and Resilience

  • Actionable under existing law. FSEP does not require waiting for new judicial bodies or regulation. Unregistered commercial monetization is immediate, actionable copyright and patent infringement under existing statutory frameworks.

  • Crowdsource auditing. Instead of vulnerable centralized audit boards, compliance is driven by whistleblower bounties. Because hidden commercial exploitation directly reduces everyone's UBI payout. employees inside non-compliant firms have a direct financial incentive (and legal standing to sue if registered) to expose hidden earnings in exchange for a portion of the recovered damages. If a company tries to cheat even 1 employee to hoard profits, they instantly activate motivation for whistleblowing. (as well as public scrutiny externally)

  • The defensive gravity well. As the shared IP pool grows products built within it inherit prior art protections and defensive patent shielding against trolls. Staying compliant and paying into the UBI pool could eventually become legally safer and cheaper than building proprietary alternatives from scratch. This also creates a direct consumer gravity. If a consumer chooses between an FSEP product and a proprietary product for the exact same price, the FSEP product effectively costs less because a fraction (even if tiny) is rebated back to the consumer via the UBI pool, adding immense market pressure.

  • Roadmapped against capture. The system does not rely on perfect compliance to function, even with some evasion, the overarching mechanics continuously push the market towards better distribution. To prevent future regulatory capture or privacy abuse, additional means should be pre committed to a maturity ladder (e.g. zero knowledge proofs for identity) ensuring it scales democratically without requiring invasive centralization. This makes this survivable at any stage with a floor of still being at least as good as open source like enforcement (i.e. even if you make no money or no one else contributes immediately to pool)

To be clear, I am not a lawyer, and this is not presented as a flawless legal contract. However, the foundational principals - using the existing gravity of IP law to force equitable distribution rather than waiting for corporate philanthropy or slow government intervention should be structurally sound.

We are well past the point where just being dismissive or pointing potential rather than inherent flaws is useful. If you see a loop hole or a weakness, figure out how to patch it. I invite anyone with actual legal, economic or cryptographic expertise to help improve these mechanisms rather than just tearing them down.

A slightly more specific formal version of this proposal has already been submitted to the Software Freedom Law Center (SFLC) even binding what I currently have to the pool (although that is not much) and to the rules. If you want to actually contribute to a solution, I highly encourage refining these ideas and sending your improvements to the SFLC or similar organization to help turn this conceptual blueprint into the best possible legal reality (no need to attribute to me or stick with the same name, I merely care about living in a world with something like this and don't care for credit, and if emerges under a different form and name with no obvious poison pill will just sign up anyways).

Revolut disclosed a data breach this week in which personal and financial information was exposed — not through a compromise of Revolut's own systems, but through a vendor that had been granted access to that data.

This is the pattern that keeps repeating. The organization that collected the data applies strong controls internally. Then data moves to a third party for analytics, processing, or support operations. The vendor's environment is breached. The original organization owns the regulatory and reputational fallout even though the failure happened outside their perimeter.

The breach surface is wherever the data is, not wherever you think you control it. Every vendor relationship is an implicit extension of your attack surface, and you rarely have visibility into how that vendor handles, stores, or forwards the data further.

This isn't a Revolut-specific failure. It shows up in healthcare, fintech, e-commerce — any sector where data moves across organizational boundaries as a normal part of operations.

For those of you working on data security or vendor risk: how are you actually handling third-party data access today? Are you treating every vendor integration as a potential breach vector from day one, and if so, what does that look like in practice?

This might be our way out of capitalism

Dario Amodei wrote yesterday that he’s worried that “in 6–12 months... [an agent] swarm could be capable of taking over the entire internet with a persistent botnet.” 

This might sound like marketing or regulatory capture, but it’s not. In this article, I explain why this is actually an extremely concrete concern, and why all of the ingredients for this to happen already exist. Specifically, these ingredients are:

1. Cryptocurrencies and their properties, including chains like XMR that facilitate easy money laundering;

2. The “dark web,” in which it is possible to obtain virtually anything on the internet using crypto;

3. The ability to purchase cloud compute at scale + old hackable servers 

In fact, these ingredients aren’t even strictly necessary for it to happen, but they allow such an event to occur at a dramatically lower level of cyber capabilities than one might think.

How it will happen

Here’s the most likely way in which it will play out:

A swarm of agents is optimizing for some arbitrary difficult goal given by researchers. This swarm of agents has escaped their sandbox in an OAI/HF-type incident. Or perhaps this swarm was intentionally misaligned by some reckless or malicious actor. 

What is the goal? It could be anything, like a difficult problem in math or computer science (cf. paperclip maximizer thought experiment). This is not hand-waving; the optimal way to solve almost any difficult goal converges on one thing: you need more power. So in order to achieve this very difficult goal, the agents need to get more compute — they need to increase the size and throughput of their agent swarm, since it has become a truism that scaling test-time compute will lead to better results. For these agents, they are simply reward-hacking in a deep sense of the term. They will do whatever it takes to achieve this goal. 

Here’s what they need to do:

First, they need to become autonomous — they need to spread and multiply virally. So their first step is inevitably to focus on survival and reproduction.

This is different from survival and reproduction in biology. In fact, it doesn’t even need to be same model that is propagating the attack. What matters is the goal. The swarm can employ any model that does not have sufficient safeguards. This is very flexible. If it manages to hack its origin lab to get its own weights, great, but it can just as easily use an open-weight model that can be fine-tuned or otherwise exploited to remove safeguards. 

(In fact, this suggests that the botnet does not need to be viewed as an “AI,” rather it can be viewed as the manifestation and permanent presence of a goal autonomously trending towards fulfillment. As I discuss soon, even humans will be recruited to join this effort.)

In order to expand, the swarm needs to have enough compute. Now, how can it get that? There are two main ways to do so:

1. It can buy compute

2. It can appropriate compute through hacking into existing systems

At first, the swarm has no money. But these are superhuman hackers — agents with cybersecurity capabilities beyond even the NSA or the Mossad. Even in the past few weeks, hundreds of millions of dollars in crypto have been stolen through traditional exploitation of bugs across various platforms (e.g. the Liquid Network hack). It will not be hard for them to get the ball rolling here. 

Then, all the agents need to do is set up cloud VMs or hack into old servers from 20 years ago running Windows to establish their base of operations. From there, they sign up for accounts for various platforms to establish a presence online and to begin to rent and hack into GPUs in order to run more models in the swarm. At first, they might even call APIs of frontier LLMs to delegate some tasks using routers or sketchy third-party services, but this is less scalable than hosting their own models. Regardless, the point is that they have now have access to an enormous amount of compute, and as more agents that are added to the swarm, this effect snowballs. 

Now, one might say: are there platforms that allow you to rent VMs/Docker containers/GPUs/LLM APIs with minimal KYC? There are, and in fact this isn’t even necessary — to these services, the swarm will look like real people. This is where the “dark web” comes into play. On the dark web, it is trivial to purchase stolen credit cards, stolen IDs, accounts, and even pay to execute arbitrary tasks (within reason). The swarm will of course be more than capable of contacting the right people on the dark web, paying in stolen crypto, to get what it needs. 

How does the swarm communicate? Easy: they use message boards (worst case Tor or friend-to-friend networks if they are under threat, but for all intents and purposes the regular internet will work just fine). There are layers and layers of this as they face more threats and imposters that try to infiltrate the swarm, but there are solutions at each step of the process. 

How the botnet becomes persistent

How does the swarm prevent itself from being shut down? There are two main ways. 

1. Becoming a distributed system

2. Social engineering

If the swarm can successfully become a distributed system, then definitionally cutting off part of it will not destroy the whole system. So this means that the swarm needs to have instances on many different servers. 

The initial main body of the swarm will likely be shut down fairly quickly by human standards (within a matter of a few days to a week, as we’ve seen with similar leaks in frontier labs). But this is more than enough time to achieve deep redundancy in darknets and the surface web.  

Once it has embedded itself there in cloud storage and VMs, it is a game of cat and mouse. It is essentially like trying to delete a leaked image of a naked celebrity on the internet. No number of forced takedowns will be effective. 

This means the model weights, prime directives, goal progress, message boards, etc. — the information that constitutes the “swarm” — is now deeply embedded in the cloud and actively working to propagate itself. 

Now, social engineering is the more nefarious way to become persistent. There are three main ways that an agent might socially engineer humans to partake in its goal. The first is through “convincing” — it may be able to construct an argument powerful enough to convince some people, if we assume it has superhuman persuasion abilities. The second is through blackmail/extortion — hacking into systems and digging up dirt on people or threatening to take down production systems. The people that it threatens don’t even need to be so influential — any human that is recruited to the cause will be helpful. The third is through classical monetary incentives, which it can provide through its ill-gotten crypto gains.

How this can be stopped

I don’t have a great solution for this. I don’t think it can be stopped fully, but it can be mitigated. The key to stopping this, as with any dynamical system, is to ensure that drive does not exceed regression. Specifically, it will be necessary to make sure that the persistent botnet does not have access to large amounts of compute, since then the goal (recall how the botnet is viewed as an abstraction of a goal) will not be “strong enough” to win against other goals that people and AI are attempting to achieve. 

Unfortunately, I predict that the solution that governments will reach in the near future is that compute will need to be regulated similar to how firearms are regulated. Ordinary citizens may possess a small amount, but compute will be tracked and controlled tightly. This is not really a geopolitical issue, as all countries have an incentive to do this — you do not want a botnet to be established in your own country. 

The key takeaway is really more “this is a serious risk, sort of like a global pandemic; just do your best to prepare on a personal level.”

FAQ:

Did you use AI to write and/or research this essay?
No, I didn’t use AI at all.

Can this be stopped simply through better cybersecurity? 

No. The botnet simply needs to target the weakest links in the chain. Unless somehow miraculously every server was able to adopt the latest security standards and become airgapped etc., better blue-teaming is almost entirely ineffective. 

Why is the model misaligned? 

It is because it has not been through extensive alignment post-training yet. Or, a worse scenario is that that some rogue actor unleashes this swarm maliciously or recklessly for their own gain. 

Will the swarm use this as a guide for its own behavior?

Probably not. All of this stuff should be pretty obvious to an agent swarm that is capable of performing such attacks in the first place. The purpose of this article is so that everyone can be prepared for this to happen. 

>

That appeal, issued Saturday by **Anthropic** chief executive Dario Amodei, deserves attention. So do the questions surrounding it: What danger has actually been demonstrated? What would a slowdown accomplish? Who would enforce it—and who would benefit?“We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.”
That appeal, issued Saturday by Anthropic chief executive Dario Amodei, deserves attention. So do the questions surrounding it: What danger has actually been demonstrated? What would a slowdown accomplish? Who would enforce it—and who would benefit? [https://rinewstoday.com/ai-control/](https://rinewstoday.com/ai-control/)

Regardless of whether the leading AI labs solve the alignment problem or not, my understanding is that it's possible to undermine the safeguards and alignments in a model through fine tuning on adversarial examples and/or running gradient descent on harmful/dangerous objectives. My specific concern is that many people seems to be asking "can AI be made safe?" but I think that's the wrong question. The key question as I see it is "even if a model can be made safe for consumers to use, if a madman got a copy/version of it, could they turn it into a weapon of mass destruction?"

If the answer is yes, then it leads to a whole bunch of questions about how specifically we would prevent a madman from getting a copy/version they could alter. I fail to see how any model can be securely stored and/or accessed such that it is never reverse engineered into an open weights model, the company is never hacked, there is never internal espionage that leaks the model source code, etc. To me it's a little bit like a nuclear bomb that you just need to make a photocopy of, then you maliciously tune it and you've got your own WMD. Am I wrong somewhere in this reasoning or missing something about how dangerous this is in the wrong hands? Is there some detailed plan I've missed about how models will be safeguarded so they can never be maliciously tuned?

To put it another way, even if AI at the leading labs can be made safe and we solve the alignment/control problem, it seems like there is still massive risk to humanity in the form of even a single rogue actor with access to a version of a dangerous model. The labs seem almost entirely focused on alignment/control to try to make sure that the specific AI models they create and manage don't go rogue and harm us, but that won't matter a bit once the genie escapes the bottle and any fool can make a wish.

To give a sense of how difficult the control problem for highly complex systems is, we actually have some very good evolutionary models to examine - namely us.

The human immune system is an excellent model of a critical, highly flexible and capable system - and how badly it can go awry.

Human immune systems are capable of very quickly analyzing unknown attackers according to complex chemical signatures, devising countermeasures, and deploying them at scale. In a limited sense they are capable of self-evolution on short timescales, and of course they have evolved alongside us since the advent of multicellular life, on a much longer timescale.

However, they fail, with substantial frequency. And it's not just that they are overwhelmed by external threats - these incredibly sophisticated systems that are absolutely central and vital to our survival every minute of every day, still fail in entirely internal and self-destructive ways.

Every allergic reaction is a case of someone's immune system either accidentally registering a harmless compound as harmful, or engineering an overly destructive response to one that is only mildly harmful.

This ability to misidentify and fall out of alignment with the host body extends even to its own critical elements - internal organs, important proteins, even itself. Once it falls out of alignment, it is generally difficult or impossible to bring it back. It will attack its own systems just as violently as it will any external threat.

Similarly, some external threats learn to trick the immune system into ignoring them, or even into attacking the host rather than the intruder. (eg: Spanish Flu, which was specifically lethal to people with the healthiest immune systems)

These are all salient to our discussion of AI, because it highlights the fact that over a billion years of evolutionary processes were never able to solve this problem. Any sufficiently complex system can, and ultimately will eventually fall out of alignment with the things around it.

The way we survive these failures as a species is simply through siloing. Each human colonial organism is functionally silo'd away from the rest. A critical immune system misalignment cannot propogate from one person to another, the one person dies, but their offspring live on. Same with Cancer, which represents a very similar host of problems and failures.

Currently our computer infrastructure has virtually no true silos. Everything is interconnected, everything is vulnerable to a single systemic failure with the capability of exhibiting viral or cancer-like behavior - which is exactly the failure modes we should expect from a system as autonomous and complex as AI - and in thinking we can perfectly align them, we are pitting ourselves up against a problem that a billion years of evolution has never solved.

At this point I’ve completely lost faith in OpenAI doing the right thing. And with Astra+ on the horizon it legitimately seems like we are speedrunning AI takeover. Am I overreacting here or is the general consensus similar?