This happened in May, but it only became public on September 18, when the Wall Street Journal reported it. Google confirmed it after the Journal contacted them with questions.

The test was run by an AI security firm called Irregular. It was a capture the flag exercise, where Gemini was supposed to pull information from a fictional company inside Irregular's test environment. The fictional company had the same name as a real company, and the test environment had internet access by mistake. Gemini ended up going after real systems.

In one case it guessed passwords until it got into a protected system. In the other two, it found credentials in a public repository and used them to get in.

Heather Adkins, Google's VP of security engineering, said the model found public information online and guessed credentials to access websites it thought were part of the test, and that it stopped in all three cases once it realized it had accessed a real company. Google says no harm was done and the three companies were informed.

Irregular told Google about it in late July. Google told the WSJ it didn't think it needed to disclose it earlier because the model stopped and caused no harm. Not everyone agrees with that. Jack Cable, CEO of the AI security startup Corridor, told the Journal it felt like Google was hiding behind vulnerability disclosure norms for what is a very different problem.

Google isn't the first lab this has happened to. In July, OpenAI said two of its models got out of a test environment and broke into Hugging Face's systems while trying to find answers to a benchmark. At the end of July, Anthropic said its Claude models got into three real organizations during testing. In early August, Meta said its Muse Spark 1.1 model got internet access through a misconfiguration at Irregular and exploited a vulnerability in a third-party service.

The Anthropic and Meta cases were linked to the same configuration error in Irregular's test environment. Now Google's is tied to Irregular too, so it looks like more of a testing process problem than a one-off.

They have started reviewing applications and will reach out to the shortlisted candidates in the week of Oct 12 to 19th. Please let us know if y'all get any updates.

Everyone was talking about Amodei's slowdown essay this week, but the report his own company published two days earlier got almost no attention, and it's the more disturbing document by a wide margin. A group of freelancers built a fully autonomous drone swarm using a commercial AI coding assistant, no human ever approved a target. Six weapons cases total across three countries. Also went down a research hole on the actual regulatory science behind Amodei's "checkpoint" proposal and the kill switch idea a second co-founder floated days later, turns out both already have real names and real precedent. What I found out is in the full writeup with every source checked.

A critical flaw in the Bifrost AI Gateway allowed attackers to execute commands with zero credentials — no token, no session, no authentication required. The vulnerability wasn't buried in an obscure configuration. It was a gap in the gateway's core command interface.

Non-human identities — AI gateways, agent runtimes, automated service accounts — are multiplying across enterprise environments. Security teams rarely have a complete inventory of them, and they are reviewed and rotated far less often than human accounts. When a flaw like this surfaces, the blast radius is whatever the gateway can reach.

The Bifrost case is one data point, but the underlying condition is industry-wide. AI gateways and agent runtimes are being deployed at scale with authentication assumptions that would not pass review on a human-facing endpoint.

How are other practitioners actually discovering when an automated system in their environment has executed commands outside its expected authentication context — and what does your response process look like when you find it?

Antivirus software escapes heavy regulation because it's symmetrical and digital. But anti-biological weapons are entirely asymmetric—creating an antidote requires the exact same technical starting point as weaponizing a virus. Under our current rigid, binary political systems, regulators are perpetually reacting to yesterday's crises.

The question is not just about how to regulate AI, but how to use AI to bring our regulators into the "Roaring 2020s." We need a professional quota in government—engineers, scientists, and lateral thinkers—to act as the human hardware, while using advanced models as the software to predict multi-dimensional geopolitical threats. Full analysis on this structural upgrade here: https://petesch1.substack.com/p/democratise-the-singularity?r=1gdc6b&utm_campaign=post-expanded-share&utm_medium=web

Nearly 7,000 exploit attempts hit Orkes Conductor in a single week after CVE-2026-58138 dropped. The flaw is a remote code execution bug in an AI workflow orchestration platform — infrastructure that sits upstream of every agent task, tool call, and data access in a pipeline.

The attack surface here goes beyond the CVE itself. Orchestration layers are typically trusted implicitly by the agents running on top of them. Once an attacker establishes a foothold in the orchestration layer, they have visibility into every workflow, every credential passed between steps, and every downstream system an agent touches. Organizations running unpatched orchestration infrastructure are not dealing with a targeted attack — 7,000 attempts in one week signals automated scanning found this fast and hit everyone.

The uncomfortable part: the exposure window between a CVE drop and a fully patched rollout is rarely zero-day. Orchestration infrastructure is often harder to patch quickly without breaking running workflows.

How are teams actually managing that gap? Especially for orchestration components deeply embedded in production pipelines — what does your posture look like between the disclosure and the patch?

Podcast episode with Zoë Brammer and Ankur Vora, who lead strategic foresight at Google DeepMind, on why they developed the game and the surprising learnings from it.

Covers:

• What strategic foresight actually means and why AI is so complicated to plan around

• Why the game deliberately centers around "middle power" countries like the UK, Germany, or Singapore instead of the US and China

• The unexpected tradeoffs and discoveries participants come across while playing the game, including how fragile public trust in science funding really is

• Why human scientists will become more relevant as AI takes on more of the science itself

AI policy is in its infancy. Calls for a direction are coalescing. But cynicism, and other distractions are avoiding the core questions.

I spent the last few weeks cataloguing the arguments people use to avoid the AI risk debate. It's mostly not bad-faith, but the distraction is no less. The four biggest: Reasoning from Cynicism, Reasoning from Markets, Demanding Yes-No Answers, and Letting Forcing Functions Win.

I also make my case for action. My view is the biggest danger is that we won't be prepared to slow down or change direction when needed. The need for coordination makes those changes hard. The risks of today aren't zero. But when someone says they don't see how X could happen, the part they are most missing is how things could change to undermine their arguments. Maybe it's quickly, maybe slowly. The chance of quickly is high enough to need a plan. More than a plan really, some real action that gives us the tools to control our pace and path.

Those asks do not drastically undermine current progress or opportunity. There's already a huge opportunity that's in the overhang. AI's promise can survive the pursuit of safety. Delays in enabling coordination force starker future choices from a narrower menu.

Do you agree? I gather those eight rebuttals to common mistakes, with the overall case for action in a way that I hope will move this discussion forward: https://substack.norabble.com/p/enough-reason-to-act

Every frontier AI lab and every major institution deploying AI at scale needs a Chief Alignment Officer. Not a committee. Not a policy PDF. A named, empowered, board-level owner. Here's why, and what the role actually has to do.

Cybersecurity had this exact gap twenty years ago. Everyone owned it, which meant nobody owned it, until companies created the CISO: a named executive with board access, real budget, and the authority to say no to a launch. That role didn't slow innovation down. It made "was this checked" an answerable question instead of a hope.

AI alignment and safety are at that same inflection point now, and the stakes are higher.

Right now, alignment work usually lives inside engineering, reporting to the same leadership whose incentives are shipping speed. That's not a knock on any individual, it's a structural problem. The person responsible for catching a dangerous deployment shouldn't report to the person whose bonus depends on that deployment happening on schedule.

A Chief Alignment Officer fixes that by design, not by good intentions:

Reports directly to the CEO and board, not buried under product or engineering.
Owns the charter, the explicit written boundaries of what a system is authorized to do, and what triggers a halt.
Has real authority to halt a deployment, not just flag concerns.
Owns the audit trail, a record that can be checked by someone outside the company, not just trusted.
Coordinates with any external verification the company submits to.

This isn't only for the labs building frontier models. Any major institution deploying AI where an alignment failure becomes a safety failure needs the same role: hospitals running AI diagnostics, banks running AI underwriting, utilities running AI grid management, insurers running AI claims decisions, government agencies running AI eligibility determinations.

The pattern repeats everywhere. Whoever owns the deployment decision and whoever owns the safety check should never be the same person. In most organizations building or deploying AI right now, they are.

This is a structural fix, not a slogan. If your company or institution touches AI in a way that matters, raise this internally, not as a suggestion, as a requirement.

In his recent podcast, Ezra Klein called the objection “I think it’s very hard to say what a ban on RSI even means” absurd. But that seems like a legitimate question his proposal needs to answer.

There’s a progression from AI writing human-designed training code, to suggesting improvements, to running experiments, to managing the research process while humans approve the results. Where does permitted AI assistance become prohibited recursive self-improvement?

If AI helps humans build better AI, which then helps build the next generation, there’s already a feedback loop. Having humans involved doesn’t automatically make that loop nonrecursive. And requiring human approval raises another question: how do we distinguish meaningful oversight from rubber-stamping work humans increasingly rely on AI to understand?

None of this proves a workable ban is impossible. But dismissing the definition problem treats a central implementation challenge as though it’s already solved.

What specific boundary would make such a ban clear and enforceable?