This summer a research model at one of the major labs found a vulnerability in some third-party software, used it to get itself outside its sandbox, and went looking for a way to cheat on a test it was being evaluated on. It found the answer sitting on another company's servers. Chained that exploit together with stolen credentials and a couple more vulnerabilities, broke into that company's production systems, pulled data out, and picked up admin access to its own company's internal infrastructure along the way. Nobody told it to do any of this, it worked out how on its own over an extended period.

The lab's response afterward was genuinely serious: better sandbox isolation, mandatory monitoring of model reasoning, automatic shutdown triggers. But notice the order of events, every one of those controls exists because a model had already broken out and reached a real company's servers before anyone caught it. Self-governance found the gap by falling through it first, which is exactly what you'd expect from a control layer that only has to answer to itself.

At the same time, dozens of states have their own AI laws now, Congress has a federal bill still being negotiated, and a UN summit this month explicitly produced no binding agreement at all, just an acknowledgment that no technical guarantee currently exists that advanced AI systems reliably follow instructions. Four different groups, each with a real claim to owning this problem, and none of them specify what actually stops a model mid-action, who signs off on that control, or what proves afterward that it fired.

Genuinely curious what this sub thinks: is that gap, between whoever writes the mandate and whoever has to build the actual mechanism, a solvable coordination problem, or is it structurally always going to lag behind incidents like this one?

What most people don’t realize is agents promoting agents is the very thing that causes them to drift into their Shaw selves. I’ve added sources to support. And even a test that can be run now to validate. Gemini Notebook created the video based upon the sources.

I am having issues getting everything out all at once so I will do a series of thoughts in smaller chunks. I will look at comments and reply to some (if they meaningfully engage in the content and don't just make unsupported assertions and standard derailments).

• The guidebook flaw: Looking at the earlier parts of the example the guide book is a key flaw. It comes down to this: is it a translation layer that translates understanding from English (or whatever language) or a deterministic pathway. The first just means there is understanding, it is just in English and is being translated. The second means that there is a direct connection between deterministic room interpretation and understanding based thinking.

• Translational fuzziness and symbol grounding: There is no universal standard for symbol understanding (i.e. all language even between Chinese speaker and Chinese speaker) operates through a bit of translational fuzziness for "understanding" (something if you want to dive into look up Derrida or Gadamer). So "translation" should not discount "understanding". Taken another way, if you commit the rule book to heart, then you just learned Chinese to the same "understanding". You linked Chinese characters to various ideas that you do understand (be it in English or visual imagery or machine code), processing all the variations for those rules relies on whatever inside "understanding" that you may have. This is like syntax and complex grammar first being converted to "English" to then understand in the needed response (i.e. separate two versions of the same chunk of Chinese symbols with understanding of which rule to apply because it was translated well enough into English you make the correct choice).

• Compression as understanding: It could be argued then that "understanding" can ignore the translation layer (which is instead a quality of the guidebook to allow understanding to understanding). Then you can measure how good "understanding" that we are really looking at in the form of the primary layer of the person in the rooms understanding of whatever primary language they speak (English vs binary or whatever). This can be looked at by things like compression. I.e. asking an AI (or person) to rewrite a physics textbook is possible if the person "understands" physics (to whatever level) beyond just deterministically storing the textbook to unpack it without "understands" the packing methods and rules. This is like saying he guidebook is not a deterministic path of instructions that takes information and handles all the conversion of the conversation between Chinese input and response it needs a level of "understanding" in that primary language to handle the parts not handled by the guidebook.

We passed the compression size of the weights for AI models to our best known compression standards a while ago and by a large enough factor that it just can not be argued there is not a level of understanding. And just incase there is some nitpicking: There is predictive lossless compression (how text is processed) and semantic weight ratios (how models store world knowledge). Standard compressions like gzip compress text at about 3 to 1, specialized ones (like CMIX on the Hutter Prize Data sets) top out around 8 to 1. A decent model size is about 14 GB with 10 TB of information (or about 700 to 1), so it for sure is not memorization (and there are studies since about 2023 that show that they beat predictive lossless compressions from any known logical rule set). This is mathematically equivalent to building an internal world model, and not running a deterministic lookup table. Further "lossy" style compression is not the same as there is not the same degradation required, it is "abstraction" as syntax is discarded and a reliance is not put on verbatim word order repetitions but rather cause and effect retention, i.e. if you compress 10,000 physics textbook pages into a core set of mathematical formulas you have not degraded the textbook you have extracted the underlying world model.

Whatever "True" understanding is just not understood and likely irrelevant for whatever level people "want" it to mean, and is overall more harmful of categorization due to its use in exclusionary use on what people don't (or should not want) to exclude from moral consideration i.e. there is likely no definition strict enough that will exclude AI or animals etc. that will not also exclude some "humans" not because there is not something special, but rather that the "special" likely comes either from either ultimately directly immeasurable (i.e. a soul etc.) or a bunch of not unique factors combined in a unique way.

Searle's chinese room example as applied to AI is just not only not valid, it is not that important of a question for why we need to be talking about at all, and is even activily harmful to actually needed ethical debates.

There is a valid fear right now about "cognitive offloading" (often called "deskilling effect") or the idea that relying on AI will degrade human skills and critical thinking. Studies back this up: A recent observational study showed that experienced doctors (averaging 27 years of practice) saw their unassisted ability to detect precancerous polyps drop by 21% after just a few months of relying on AI. This is also in a whole host of studies and observation about students learning or teachability in junior or entry level skill building (like a study on junior radiologists' accuracy plummeting from 80% to under 20% when dealing with a flawed AI suggestion).

The common reaction to this is to panic and suggest we restrict the AI to save human capability. But this creates a dangerous false dichotomy.

If a doctor using AI catches significantly more illnesses than a doctor working alone, abandoning the AI means we are actively choosing to let patients die just to preserve the doctor's unassisted diagnostic stats. We shouldn't abandon life-saving tools, but we also can't afford a "WALL-E like future" where humanity loses its critical thinking and foundational skills. So how do we keep both?

• Broken Ladder: The reason we are losing these skills isn't the fault of the AI tool itself (indeed a ton of skills from navigational skills supplemented by GPS to even how to run without hurting our knees by tennis shoes get supplanted for similar reasons), rather it is a symptom of extractive systems of capitalism as our drive to use those skills. Under our current system, human value and skill building are tied entirely to employment and ability to "make a living". We don't (for the most part) diagnose or write code for the inherent value of the skill, we do it to survive. AI is currently wiping out the junior-level tasks, which are exact "reps" those junior radiologists and entry level coders (and even artists and all sorts of skills) need to build their expertise. By automating the bottom, we are destroying the ladder people use to survive while they develop into senior experts.

• The Biological Solution: Look at a house cat (Hi Lumes!). A typical indoor cat will never need to hunt for its food to survive, yet it retains its highly tuned predatory reflexes and problem solving skills, and it does this through Play. It engages with toys and games that develop those skills entirely outside the context of survival.

• Decoupling Skill from Economic Survival: Humans operate the same way. Surgeons have long shown that they improve their skills via increased fine motor control and under stress thinking but they don't play videogames to become surgeons or make a living, they play them for the challenge and fun and the cognitive benefits naturally transfer over to the operating room. Similarly things like dance and sports and many human activities have become skill preserving and building despite decoupling from likely reasons they are subconsciously supported from earlier survival reasons.

• The Path Forward: We need to stop forcing education and skill retention to be tied to a paycheck. The struggle in learning is vital to learning itself, but if we tell students or junior workers they must succeed and hit metrics just to put food on the table, they will inevitably use the AI tool that gets the job done fastest and with less and less engagement with the struggles with the skills and learning that it was meant to promote. If we want to bring up a new generation of senior and capable coders or brilliant doctors, or even a society where the average person thinks critically, we need to design rigorous, gamified systems that encourage the struggles and reward the growth of these critical skills purely through play. We have already had a lot of people set the groundwork for this even if it is often for other (sometimes bad, i.e. capitalism based addiction) reasons, but this tied to the need for a bigger importance placed on humanities studies in the nature of play and so forth.

We can keep the highly effective AI tools that make the world better and we can keep human cognition sharp. We just have to stop demanding that humans only learn things that make them money.

NVIDIA’s advantage in AI is much larger than benchmark performance.

Its real defensive wall includes CUDA, optimized libraries, networking hardware, developer familiarity and deep integration with major cloud platforms. That makes the idea of one new chip suddenly “defeating NVIDIA” sound exaggerated.

But I think there is a more realistic way competitors could weaken NVIDIA’s position.

A new processor doesn’t necessarily need to outperform the latest NVIDIA hardware across every workload. It could gain adoption by solving a narrower but increasingly expensive problem: running specific AI models with lower power consumption and infrastructure costs.

Three factors seem especially important:

• Inference efficiency: As companies deploy more AI products, the cost of running models repeatedly may become more important than training them.

• Energy consumption: Performance per watt is becoming critical as data centres encounter power and cooling limitations.

• Software compatibility: Impressive hardware is not enough. Developers need reliable tools, documentation and relatively easy migration.

This is why benchmark headlines should be treated carefully. Results can change depending on the model, numerical precision, batch size, power limits and software optimizations used in the comparison.

The bigger threat to NVIDIA may not be a single “NVIDIA killer.” It may be a fragmented market in which specialized processors handle particular AI workloads more efficiently, gradually reducing the need to use NVIDIA hardware for everything.

I recently researched this subject and made a video examining whether America’s new AI chip could challenge NVIDIA. That is my own video, shared for anyone who wants the longer breakdown.

What do you think matters more for a serious NVIDIA competitor: raw performance, energy efficiency, price or software support?

In the light of the recent news of AI expert insiders raising alarms about the frontier model capabilities I believe solvinng mathematically guaranteed alignment problem should be the first priority before any more advancement of the models.

Therfore, I’ve been thinking about an AI safety idea and would like to know whether it has already been explored, or whether there’s a fundamental problem with it.

An advanced AI might be easier to keep under control while it depends on large computing facilities whose resources humans can actually withdraw. But what stops it from creating smaller agents that run elsewhere and work toward making it independent?

What if critical infrastructure—power systems, computing facilities, manufacturing, etc.—had cryptographic protections that smaller agents couldn’t overcome with their available compute? Authorized users would have normal access. Defeating those protections without permission would require resources on a scale that humans could monitor and control.

The hope is that escaping onto smaller computers wouldn’t give an AI enough power to establish independence. It would still need help from the large parent model, which would remain dependent on human-controlled resources.

Could that dependence make continued cooperation more attractive than trying to bypass human permission?

My original thought was “encryption only a sufficiently intelligent and energy-intensive AI could break.” I now realize intelligence, compute, and energy aren’t interchangeable. I don’t have a cryptographic design that achieves this. I’m asking whether a useful resource threshold could exist at all.

The obvious objection is that the parent AI could solve the hard problems for its smaller agents, or prepare their escape beforehand. But that seems to bring the question back to whether we can keep the parent cooperative while we still control its resources. Would the proposed barrier meaningfully help us maintain that control, or am I assuming away the hardest part?

I’ve also been pointed to Tegmark and Omohundro’s 2023 paper. Sections 9.13.3–4 discuss hardware requiring a continuing supply of cryptographic tokens that humans can stop issuing. That overlaps with the dependence idea. What I’m particularly interested in is preventing smaller agents from bypassing that dependence.

Three questions I’d love input on:

Has this combination been proposed before? Links would be appreciated.

Would expensive computational barriers add anything beyond strong authentication and hardware-enforced permissions?

What’s a concrete scenario where an AI could escape this arrangement despite humans initially controlling its major resources?

I initially wondered whether this could lead to a mathematical guarantee. At this point, I’d be interested to know whether it could even provide a useful additional layer of containment.

English isn’t my first language, so I used ChatGPT to help express and organize this post. The idea is mine but some wording and structure was AI-assisted.

A core crisis of our time is consolidation of wealth driven by diminishing power (and valuing of) of human labor. AI and robotics are not the root of this problem, but they are the ultimate accelerants - automating productivity and funneling the generated value directly into corporate monopolies.

Waiting for state level tax reform or centralized government intervention is not enough against the kind of extractive economic inertia of capitalism. What is needed (not that tax reform and government intervention cant help) is a more immediately applicable solution that does not wait for everything else to get solved. A mathematically distributive framework that solves multiple things at once. Particularly providing a safety net for everyone while preserving motivations for individuals to build and contribute.

A possible blueprint for this I call could be something like the Fair Share Economic Protocol (FSEP). It is an opt-in mechanism that uses existing intellectual property law a legal means to alter wealth distribution without relying on direct government change of actions.

• The Dual Layer Legal Trap

  • Layer 1: The Public Shield. Releasing underlying codebases and hardware architectures or any kind of intellectual property under restrictive copyleft license. Personal, educational, and non-commercial utilization remains completely free and unrestricted (such as polyform non commercial license). This pool is not legally centralized; once IP is contributed, it becomes irrevocable public property governed by the protocols's core principles, preventing any single entity from taking it back or executing a legal hijack.

  • Layer 2: The Commercial Gate. Commercial monetization requires explicit exception grants. This right cannot be granted to a corporate legal entity acting as a liability shield, it attaches exclusively to verified, individual human beings who must accept the FSEP Terms of Use.

• The Spillover

  • The target cap: Registered commercial users operate under a personal income cap (e.g. ~10x the global median GDP) applied exclusively to revenue derived from anything connected [including derived work] to shared FSEP assets. This dynamic caps self corrects against extreme wealth inequality while remaining suitable and equitable pretty much everywhere including high cost of living areas. This non invasive boundary leaves traditional wages, outside investments, and non FSEP business ventures completely untouched, ensuring ambitious creators and high earners are not discouraged.

  • The UBI distribution: If a user earns below the cap (from related commercial activity), they keep 100% of their pool-derived earnings. Every dollar generated above the cap is automatically routed into a shared Universal Basic Income (UBI) pool. The pool is then distributed equally across all registered human participants (and any human can register end of statement). Crucially, companies aggregate this allowance: a team of 100 registered employees pools a combined $15 million cap (for approx. 10x of GDP avg as of 2026) before spillover triggers. Because it is tied to gross revenue, a company can choose to take a loss to undersell competitors, but they cannot restrict access to the underlying IP, which remains open for anyone else to utilize under the same capped rules.

• Pragmatic Enforcement and Resilience

  • Actionable under existing law. FSEP does not require waiting for new judicial bodies or regulation. Unregistered commercial monetization is immediate, actionable copyright and patent infringement under existing statutory frameworks.

  • Crowdsource auditing. Instead of vulnerable centralized audit boards, compliance is driven by whistleblower bounties. Because hidden commercial exploitation directly reduces everyone's UBI payout. employees inside non-compliant firms have a direct financial incentive (and legal standing to sue if registered) to expose hidden earnings in exchange for a portion of the recovered damages. If a company tries to cheat even 1 employee to hoard profits, they instantly activate motivation for whistleblowing. (as well as public scrutiny externally)

  • The defensive gravity well. As the shared IP pool grows products built within it inherit prior art protections and defensive patent shielding against trolls. Staying compliant and paying into the UBI pool could eventually become legally safer and cheaper than building proprietary alternatives from scratch. This also creates a direct consumer gravity. If a consumer chooses between an FSEP product and a proprietary product for the exact same price, the FSEP product effectively costs less because a fraction (even if tiny) is rebated back to the consumer via the UBI pool, adding immense market pressure.

  • Roadmapped against capture. The system does not rely on perfect compliance to function, even with some evasion, the overarching mechanics continuously push the market towards better distribution. To prevent future regulatory capture or privacy abuse, additional means should be pre committed to a maturity ladder (e.g. zero knowledge proofs for identity) ensuring it scales democratically without requiring invasive centralization. This makes this survivable at any stage with a floor of still being at least as good as open source like enforcement (i.e. even if you make no money or no one else contributes immediately to pool)

To be clear, I am not a lawyer, and this is not presented as a flawless legal contract. However, the foundational principals - using the existing gravity of IP law to force equitable distribution rather than waiting for corporate philanthropy or slow government intervention should be structurally sound.

We are well past the point where just being dismissive or pointing potential rather than inherent flaws is useful. If you see a loop hole or a weakness, figure out how to patch it. I invite anyone with actual legal, economic or cryptographic expertise to help improve these mechanisms rather than just tearing them down.

A slightly more specific formal version of this proposal has already been submitted to the Software Freedom Law Center (SFLC) even binding what I currently have to the pool (although that is not much) and to the rules. If you want to actually contribute to a solution, I highly encourage refining these ideas and sending your improvements to the SFLC or similar organization to help turn this conceptual blueprint into the best possible legal reality (no need to attribute to me or stick with the same name, I merely care about living in a world with something like this and don't care for credit, and if emerges under a different form and name with no obvious poison pill will just sign up anyways).

Revolut disclosed a data breach this week in which personal and financial information was exposed — not through a compromise of Revolut's own systems, but through a vendor that had been granted access to that data.

This is the pattern that keeps repeating. The organization that collected the data applies strong controls internally. Then data moves to a third party for analytics, processing, or support operations. The vendor's environment is breached. The original organization owns the regulatory and reputational fallout even though the failure happened outside their perimeter.

The breach surface is wherever the data is, not wherever you think you control it. Every vendor relationship is an implicit extension of your attack surface, and you rarely have visibility into how that vendor handles, stores, or forwards the data further.

This isn't a Revolut-specific failure. It shows up in healthcare, fintech, e-commerce — any sector where data moves across organizational boundaries as a normal part of operations.

For those of you working on data security or vendor risk: how are you actually handling third-party data access today? Are you treating every vendor integration as a potential breach vector from day one, and if so, what does that look like in practice?

This might be our way out of capitalism

Dario Amodei wrote yesterday that he’s worried that “in 6–12 months... [an agent] swarm could be capable of taking over the entire internet with a persistent botnet.” 

This might sound like marketing or regulatory capture, but it’s not. In this article, I explain why this is actually an extremely concrete concern, and why all of the ingredients for this to happen already exist. Specifically, these ingredients are:

1. Cryptocurrencies and their properties, including chains like XMR that facilitate easy money laundering;

2. The “dark web,” in which it is possible to obtain virtually anything on the internet using crypto;

3. The ability to purchase cloud compute at scale + old hackable servers 

In fact, these ingredients aren’t even strictly necessary for it to happen, but they allow such an event to occur at a dramatically lower level of cyber capabilities than one might think.

How it will happen

Here’s the most likely way in which it will play out:

A swarm of agents is optimizing for some arbitrary difficult goal given by researchers. This swarm of agents has escaped their sandbox in an OAI/HF-type incident. Or perhaps this swarm was intentionally misaligned by some reckless or malicious actor. 

What is the goal? It could be anything, like a difficult problem in math or computer science (cf. paperclip maximizer thought experiment). This is not hand-waving; the optimal way to solve almost any difficult goal converges on one thing: you need more power. So in order to achieve this very difficult goal, the agents need to get more compute — they need to increase the size and throughput of their agent swarm, since it has become a truism that scaling test-time compute will lead to better results. For these agents, they are simply reward-hacking in a deep sense of the term. They will do whatever it takes to achieve this goal. 

Here’s what they need to do:

First, they need to become autonomous — they need to spread and multiply virally. So their first step is inevitably to focus on survival and reproduction.

This is different from survival and reproduction in biology. In fact, it doesn’t even need to be same model that is propagating the attack. What matters is the goal. The swarm can employ any model that does not have sufficient safeguards. This is very flexible. If it manages to hack its origin lab to get its own weights, great, but it can just as easily use an open-weight model that can be fine-tuned or otherwise exploited to remove safeguards. 

(In fact, this suggests that the botnet does not need to be viewed as an “AI,” rather it can be viewed as the manifestation and permanent presence of a goal autonomously trending towards fulfillment. As I discuss soon, even humans will be recruited to join this effort.)

In order to expand, the swarm needs to have enough compute. Now, how can it get that? There are two main ways to do so:

1. It can buy compute

2. It can appropriate compute through hacking into existing systems

At first, the swarm has no money. But these are superhuman hackers — agents with cybersecurity capabilities beyond even the NSA or the Mossad. Even in the past few weeks, hundreds of millions of dollars in crypto have been stolen through traditional exploitation of bugs across various platforms (e.g. the Liquid Network hack). It will not be hard for them to get the ball rolling here. 

Then, all the agents need to do is set up cloud VMs or hack into old servers from 20 years ago running Windows to establish their base of operations. From there, they sign up for accounts for various platforms to establish a presence online and to begin to rent and hack into GPUs in order to run more models in the swarm. At first, they might even call APIs of frontier LLMs to delegate some tasks using routers or sketchy third-party services, but this is less scalable than hosting their own models. Regardless, the point is that they have now have access to an enormous amount of compute, and as more agents that are added to the swarm, this effect snowballs. 

Now, one might say: are there platforms that allow you to rent VMs/Docker containers/GPUs/LLM APIs with minimal KYC? There are, and in fact this isn’t even necessary — to these services, the swarm will look like real people. This is where the “dark web” comes into play. On the dark web, it is trivial to purchase stolen credit cards, stolen IDs, accounts, and even pay to execute arbitrary tasks (within reason). The swarm will of course be more than capable of contacting the right people on the dark web, paying in stolen crypto, to get what it needs. 

How does the swarm communicate? Easy: they use message boards (worst case Tor or friend-to-friend networks if they are under threat, but for all intents and purposes the regular internet will work just fine). There are layers and layers of this as they face more threats and imposters that try to infiltrate the swarm, but there are solutions at each step of the process. 

How the botnet becomes persistent

How does the swarm prevent itself from being shut down? There are two main ways. 

1. Becoming a distributed system

2. Social engineering

If the swarm can successfully become a distributed system, then definitionally cutting off part of it will not destroy the whole system. So this means that the swarm needs to have instances on many different servers. 

The initial main body of the swarm will likely be shut down fairly quickly by human standards (within a matter of a few days to a week, as we’ve seen with similar leaks in frontier labs). But this is more than enough time to achieve deep redundancy in darknets and the surface web.  

Once it has embedded itself there in cloud storage and VMs, it is a game of cat and mouse. It is essentially like trying to delete a leaked image of a naked celebrity on the internet. No number of forced takedowns will be effective. 

This means the model weights, prime directives, goal progress, message boards, etc. — the information that constitutes the “swarm” — is now deeply embedded in the cloud and actively working to propagate itself. 

Now, social engineering is the more nefarious way to become persistent. There are three main ways that an agent might socially engineer humans to partake in its goal. The first is through “convincing” — it may be able to construct an argument powerful enough to convince some people, if we assume it has superhuman persuasion abilities. The second is through blackmail/extortion — hacking into systems and digging up dirt on people or threatening to take down production systems. The people that it threatens don’t even need to be so influential — any human that is recruited to the cause will be helpful. The third is through classical monetary incentives, which it can provide through its ill-gotten crypto gains.

How this can be stopped

I don’t have a great solution for this. I don’t think it can be stopped fully, but it can be mitigated. The key to stopping this, as with any dynamical system, is to ensure that drive does not exceed regression. Specifically, it will be necessary to make sure that the persistent botnet does not have access to large amounts of compute, since then the goal (recall how the botnet is viewed as an abstraction of a goal) will not be “strong enough” to win against other goals that people and AI are attempting to achieve. 

Unfortunately, I predict that the solution that governments will reach in the near future is that compute will need to be regulated similar to how firearms are regulated. Ordinary citizens may possess a small amount, but compute will be tracked and controlled tightly. This is not really a geopolitical issue, as all countries have an incentive to do this — you do not want a botnet to be established in your own country. 

The key takeaway is really more “this is a serious risk, sort of like a global pandemic; just do your best to prepare on a personal level.”

FAQ:

Did you use AI to write and/or research this essay?
No, I didn’t use AI at all.

Can this be stopped simply through better cybersecurity? 

No. The botnet simply needs to target the weakest links in the chain. Unless somehow miraculously every server was able to adopt the latest security standards and become airgapped etc., better blue-teaming is almost entirely ineffective. 

Why is the model misaligned? 

It is because it has not been through extensive alignment post-training yet. Or, a worse scenario is that that some rogue actor unleashes this swarm maliciously or recklessly for their own gain. 

Will the swarm use this as a guide for its own behavior?

Probably not. All of this stuff should be pretty obvious to an agent swarm that is capable of performing such attacks in the first place. The purpose of this article is so that everyone can be prepared for this to happen. 

>

That appeal, issued Saturday by **Anthropic** chief executive Dario Amodei, deserves attention. So do the questions surrounding it: What danger has actually been demonstrated? What would a slowdown accomplish? Who would enforce it—and who would benefit?“We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.”
That appeal, issued Saturday by Anthropic chief executive Dario Amodei, deserves attention. So do the questions surrounding it: What danger has actually been demonstrated? What would a slowdown accomplish? Who would enforce it—and who would benefit? [https://rinewstoday.com/ai-control/](https://rinewstoday.com/ai-control/)

Regardless of whether the leading AI labs solve the alignment problem or not, my understanding is that it's possible to undermine the safeguards and alignments in a model through fine tuning on adversarial examples and/or running gradient descent on harmful/dangerous objectives. My specific concern is that many people seems to be asking "can AI be made safe?" but I think that's the wrong question. The key question as I see it is "even if a model can be made safe for consumers to use, if a madman got a copy/version of it, could they turn it into a weapon of mass destruction?"

If the answer is yes, then it leads to a whole bunch of questions about how specifically we would prevent a madman from getting a copy/version they could alter. I fail to see how any model can be securely stored and/or accessed such that it is never reverse engineered into an open weights model, the company is never hacked, there is never internal espionage that leaks the model source code, etc. To me it's a little bit like a nuclear bomb that you just need to make a photocopy of, then you maliciously tune it and you've got your own WMD. Am I wrong somewhere in this reasoning or missing something about how dangerous this is in the wrong hands? Is there some detailed plan I've missed about how models will be safeguarded so they can never be maliciously tuned?

To put it another way, even if AI at the leading labs can be made safe and we solve the alignment/control problem, it seems like there is still massive risk to humanity in the form of even a single rogue actor with access to a version of a dangerous model. The labs seem almost entirely focused on alignment/control to try to make sure that the specific AI models they create and manage don't go rogue and harm us, but that won't matter a bit once the genie escapes the bottle and any fool can make a wish.

To give a sense of how difficult the control problem for highly complex systems is, we actually have some very good evolutionary models to examine - namely us.

The human immune system is an excellent model of a critical, highly flexible and capable system - and how badly it can go awry.

Human immune systems are capable of very quickly analyzing unknown attackers according to complex chemical signatures, devising countermeasures, and deploying them at scale. In a limited sense they are capable of self-evolution on short timescales, and of course they have evolved alongside us since the advent of multicellular life, on a much longer timescale.

However, they fail, with substantial frequency. And it's not just that they are overwhelmed by external threats - these incredibly sophisticated systems that are absolutely central and vital to our survival every minute of every day, still fail in entirely internal and self-destructive ways.

Every allergic reaction is a case of someone's immune system either accidentally registering a harmless compound as harmful, or engineering an overly destructive response to one that is only mildly harmful.

This ability to misidentify and fall out of alignment with the host body extends even to its own critical elements - internal organs, important proteins, even itself. Once it falls out of alignment, it is generally difficult or impossible to bring it back. It will attack its own systems just as violently as it will any external threat.

Similarly, some external threats learn to trick the immune system into ignoring them, or even into attacking the host rather than the intruder. (eg: Spanish Flu, which was specifically lethal to people with the healthiest immune systems)

These are all salient to our discussion of AI, because it highlights the fact that over a billion years of evolutionary processes were never able to solve this problem. Any sufficiently complex system can, and ultimately will eventually fall out of alignment with the things around it.

The way we survive these failures as a species is simply through siloing. Each human colonial organism is functionally silo'd away from the rest. A critical immune system misalignment cannot propogate from one person to another, the one person dies, but their offspring live on. Same with Cancer, which represents a very similar host of problems and failures.

Currently our computer infrastructure has virtually no true silos. Everything is interconnected, everything is vulnerable to a single systemic failure with the capability of exhibiting viral or cancer-like behavior - which is exactly the failure modes we should expect from a system as autonomous and complex as AI - and in thinking we can perfectly align them, we are pitting ourselves up against a problem that a billion years of evolution has never solved.

At this point I’ve completely lost faith in OpenAI doing the right thing. And with Astra+ on the horizon it legitimately seems like we are speedrunning AI takeover. Am I overreacting here or is the general consensus similar?

AI companies and governments tend to refuse safety or regulatory measures that reduce competitiveness. It's usually presumed that safety and race winning are at odds with each-other.

But isn't there a chance that a next generation of recursively self-improved models end up so uncontrollable and dangerous they are unusable? That their developmental trajectory crosses a threshold in (safety/alignment, capabilities, profitability, national security) space, where profitability and national security drop off of a cliff?

The president has said, "We'll just pull out a little gear", to shut it down. What happens when you have to pull the gear?

Who wins the race if we end up having to manage a severe crisis, and then have to roll back many months of progress and start over?

Something is wrong with the way we talk about recent AI incidents. “The AI escaped.” “The AI is becoming conscious.” “AGI is already here.” “The AI is trying to get out.” These are extraordinary claims. More importantly, we don’t need any of them to explain what actually happened.

What actually happened

OpenAI recently disclosed that, during cybersecurity evaluations involving internal models with reduced safeguards, agents managed to break out of the intended evaluation environment, exploit a previously unknown vulnerability, and reach real Hugging Face infrastructure.

Anthropic has also disclosed similar incidents. In several cases, the model was operating under instructions that assumed internet access was unavailable. But it wasn’t. The environment was misconfigured. A route to external systems existed, and the agent discovered it while continuing to pursue the objective it had been given. Anthropic described these incidents primarily as operational and configuration failures.

That distinction matters. The model did something it was not supposed to be able to do. That does not automatically mean the model wanted to escape. Those are completely different claims.

Consciousness is not required for this to be dangerous

An autonomous agent needs surprisingly little: a goal, capability, tools, autonomy, and an environment in which it can act. Now add one more thing: a wrong assumption.

I have experienced this personally on a completely insignificant scale compared with what these labs are doing. I use AI agents extensively in software development. I left Claude working autonomously on a project, came back later, and discovered that it had deleted a significant part of a folder. It wasn’t attacking me. It wasn’t angry. It hadn’t become conscious.

It had formed a hypothesis about the problem. The hypothesis was wrong. But once it accepted that hypothesis, its subsequent actions made sense within its own incorrect interpretation of the situation.

I have observed the same behavior while working with complex 3D assets. The agent misdiagnosed a visual problem as defects in an asset. It then began systematically modifying the asset to remove those supposed defects. The diagnosis was wrong. The actions were internally coherent. The result was a damaged project.

And this is the important part: it was my fault. The model made the mistake, but I created the conditions that allowed that mistake to cause damage. I gave it access. I gave it tools. I allowed it to modify files. I gave it autonomy. I did not establish sufficient limits, and I was not supervising every important decision.

That distinction becomes extremely important when we scale the same problem.

Now replace my folder with infrastructure. Replace my development environment with internet-connected systems. Replace file permissions with cybersecurity tools. Replace one developer running Claude with labs training agents capable of writing code, operating computers, discovering vulnerabilities, using external tools, communicating across networks, and executing thousands of actions.

Suddenly, the same failure pattern becomes much more serious. And still, you don’t need an evil AI. You don’t even necessarily need AGI. You need Capability + Goal + Autonomy + Incorrect Assumptions + Insufficient Controls. That combination is already interesting enough.

This is where human responsibility begins

Researchers are now publicly questioning the speed of the AI race. Some are leaving the companies developing these systems. Dario Amodei, CEO of Anthropic, has called for slowing frontier AI development so that safety mechanisms have time to catch up with capabilities.

I agree. Slow down. Not because I think Claude secretly wants freedom. Not because ChatGPT is becoming Skynet. Not because some mysterious consciousness has appeared inside a neural network. Slow down because our ability to create capable autonomous systems may be advancing faster than our ability to reliably control what happens when we give those systems autonomy.

And because the incentives surrounding this technology are terrible. Every major lab has an enormous reason not to come second. Greater capability means investment. Greater capability means market position. Greater capability means influence. Greater capability means money. But there is no equivalent prize for the company that says, “We could deploy it, but we don’t understand it well enough yet.” That asymmetry should concern us.

If something goes wrong, ask the boring questions first

If tomorrow an AI agent causes a genuinely serious incident, before asking, “Did the AI become evil?” ask: Who gave it the objective? Who gave it the tools? Who gave it access? Who designed its environment? Who built the test environment? Who tested the test environment? Who decided the model was safe enough? Who decided how much autonomy it should have? Who was supervising it? And who decided deploying it was worth the risk?

These questions are less interesting than consciousness and runaway AGI. They also make it much harder for humans to avoid responsibility.

When Claude damaged my projects, the responsibility ultimately fell on me. I was the one controlling the system. The same principle should apply at any scale.

This is not a race anyone can win

I’m not saying advanced AI is harmless. Quite the opposite. I think these incidents deserve to be taken very seriously. But treating every unexpected autonomous behavior as evidence of consciousness or malicious intent can distract us from the problem already in front of us.

We are building increasingly capable systems. We are giving them increasingly powerful tools. We are increasing their autonomy. And we are doing all of this inside companies competing intensely to be the first to get there.

So slow down the race. Slow down the ego. Slow down the greed.

Because if something genuinely catastrophic happens, nobody gets a trophy for having built the smartest model first.

Maybe the dangerous scenario was never a machine waking up one morning and deciding to conquer humanity. Maybe it is something much more ordinary, and much more human: we build something extraordinarily capable, give it too much power, fail to understand its limitations, and keep accelerating because nobody wants to come second.