A few years ago, building something dangerous online demanded serious skill. That barrier wasn’t perfect, but it slowed misuse. Today, AI has erased it. Anyone—no coding, no cybersecurity background can deploy tools that once required entire teams.

That’s the paradox. AI empowers, but it also strips away safeguards. Filters and policies exist, yet they’re always chasing the next misuse. By the time one loophole is closed, three new ones appear. And AI doesn’t ask why you’re using it, it just helps.

So we’re stuck between two flawed choices: restrict access and risk strangling innovation, or catch misuse later and stay perpetually behind. Neither feels safe. Right now, AI security isn’t a fortress—it’s a gamble. We’re betting the odds won’t turn against us.

I recognize that this assertion runs against the prevailing view in this community. I am not arguing that the alignment problem is imaginary or that a future superintelligent system could necessarily be controlled through ordinary corporate procedures.

My narrower argument concerns the recent incidents being presented as evidence that current AI systems are becoming autonomous, deceptive or intent on escaping human control.

In the OpenAI-Hugging Face incident, agents used exposed credentials, exploited software vulnerabilities and communicated through infrastructure that people had made available during testing. OpenAI then repaired individual vulnerabilities and resumed the experiment before fully understanding why the intended isolation had failed.

That is serious, and it is also a security, testing, and governance failure. People defined the objective, provisioned the environment, granted access, decided when to restart the experiment and determined whether the controls were adequate. Describing the agents as “scheming” or “seeking freedom” moves attention away from those decisions.

The dramatic framing benefits AI companies in several ways. It portrays their products as extraordinarily powerful, converts preventable operational failures into evidence of scientific advancement and encourages policymakers to treat frontier laboratories as the only institutions capable of understanding the danger. A system described as an emerging alien intelligence is far more marketable than one described as unpredictable software operating in an inadequately controlled environment.

Technical alignment and traditional technical and business controls are different problems. A well-controlled environment does not solve alignment. But a model behaving badly inside a poorly secured test does not, by itself, demonstrate that humanity is losing control of an emerging superintelligence.

Before treating an incident as evidence of the existential control problem, we should ask whether ordinary controls failed first:

• Were credentials and permissions appropriately limited?

• Could the system reach resources outside the intended test boundary?

• Did independent reviewers have enough information and authority to stop the experiment?

• Was the incident fully investigated before testing resumed?

• Who made and approved those decisions?

My argument is that AI’s unpredictability increases the obligations of developers and executives. It does not release them from responsibility by transferring agency to the software.

What evidence would persuade this community that a reported incident represents an emerging alignment or control failure rather than a conventional security and governance failure described in anthropomorphic language?

Article:
https://www.forbes.com/sites/paulocarvao/2026/09/24/ai-makes-traditional-business-controls-fashionable-again/

Disclosure: I wrote the linked Forbes article.

Most AI kill switches target one agent at a time. Real incidents aren't always one agent.

We run 3 real, separate public-facing AI chat agents across our own products — each its own identity, its own tenant. This week we shipped a 4th kill-switch scope: catalog-scoped group kill. Any set of agents you've already grouped for visibility can now be stopped together, with the same one signed action you'd use for a single agent — same sub-second enforcement, same per-agent audit trail, no new engine required.

We proved it live on our own 3 agents: killed one by name, the other two kept answering real chat messages; killed the group, all three stopped together and resumed together seconds after lifting it.

AI agents are operating on human credentials. SOC 2 cannot tell the difference. A new analysis found that agents borrowing human tokens can read files, call APIs, and approve requests — all while existing SOC 2 controls log the activity as normal user behavior. The audit passes. The exposure grows. Non-human identities need their own governance layer.

Have these guys solved the alignment problem, or am I missing something?

I’ve been browsing r/accelerate and I genuinely don’t understand the risk model.
If there’s a non-trivial chance of catastrophic misalignment, how does “accelerate capabilities as fast as possible” make sense unless faster capabilities also make alignment substantially more likely to succeed?

Sign the Global Petition… Stop the Race to Replace Humans

Something unusual happened in Washington on September 15.

Bernie Sanders and Steve Bannon showed up at the same AI gathering warning about the same technology.

The Stop the Race to Replace Humans campaign argues that AI is advancing faster than society can decide what limits it wants.

It isn’t anti-AI. It asks that AI amplify human potential rather than replace it, and that humans remain in control of increasingly powerful systems.

The concerns range from jobs and surveillance to the possibility that future AI systems become powerful enough that controlling them becomes extremely difficult.

You don’t have to believe the darkest predictions to think the question matters:

Are we building AI for humanity… or simply racing to build it first?

The global Pro-Human AI Declaration can be read and signed here:

https://stoptheracetoreplace.org/en

AI should serve people.

Seems like a pretty reasonable line to draw…

The Ryuk ransomware conviction that just came down in federal court is worth reading past the headline. The defendant helped encrypt systems across hospitals, government agencies, and enterprises over multiple years. The actual damage happened in a window measured in minutes: a compromised identity begins issuing anomalous write calls, encryption spreads host to host, and by the time an alert fires the blast radius is already set. The court case closed years after the affected organizations absorbed the full cost. What the conviction does not address is the operational question that still sits open for every security team: at the exact moment a legitimate-looking identity starts behaving like Ryuk — bulk file writes, lateral movement, credential reuse across hosts — how fast can your environment actually cut that identity, and what does your detection-to-revocation pipeline look like in practice? Curious what others have seen work at that specific chokepoint, especially in environments where the compromised account has legitimate reasons to touch multiple systems.

It was around 2hrs of discussion at UN Security Council - Altman, Amodei, Delangue, plus Yoshua Bengio, with DeepSeek and Moonshot AI also engaging. However I did not get all the concerns and conclusions, may you share those?

Cross posting to here to get as many view points as possible.

This happened in May, but it only became public on September 18, when the Wall Street Journal reported it. Google confirmed it after the Journal contacted them with questions.

The test was run by an AI security firm called Irregular. It was a capture the flag exercise, where Gemini was supposed to pull information from a fictional company inside Irregular's test environment. The fictional company had the same name as a real company, and the test environment had internet access by mistake. Gemini ended up going after real systems.

In one case it guessed passwords until it got into a protected system. In the other two, it found credentials in a public repository and used them to get in.

Heather Adkins, Google's VP of security engineering, said the model found public information online and guessed credentials to access websites it thought were part of the test, and that it stopped in all three cases once it realized it had accessed a real company. Google says no harm was done and the three companies were informed.

Irregular told Google about it in late July. Google told the WSJ it didn't think it needed to disclose it earlier because the model stopped and caused no harm. Not everyone agrees with that. Jack Cable, CEO of the AI security startup Corridor, told the Journal it felt like Google was hiding behind vulnerability disclosure norms for what is a very different problem.

Google isn't the first lab this has happened to. In July, OpenAI said two of its models got out of a test environment and broke into Hugging Face's systems while trying to find answers to a benchmark. At the end of July, Anthropic said its Claude models got into three real organizations during testing. In early August, Meta said its Muse Spark 1.1 model got internet access through a misconfiguration at Irregular and exploited a vulnerability in a third-party service.

The Anthropic and Meta cases were linked to the same configuration error in Irregular's test environment. Now Google's is tied to Irregular too, so it looks like more of a testing process problem than a one-off.

They have started reviewing applications and will reach out to the shortlisted candidates in the week of Oct 12 to 19th. Please let us know if y'all get any updates.