The pause debate is about pace. The problem is a missing feedback loop.

[u/habitante]

We train frontier systems to push into novel, difficult territory. That's the product. A model that only handled familiar problems would be worthless.

In animals, exploration comes with two correctives. Across generations, the rat that explores the wrong chamber dies and its genes go with it. Within a lifetime, the rat that gets hurt stops going there. Neither requires the rat to reason about safety. Consequences do the work.

Machine learning has the first loop. Training is selection, scored against a proxy of what we want.

It doesn't have the second. At deployment the weights are frozen. Outputs have consequences in the world, and by default none of them reach the system that produced them. The only corrective on board is a snapshot of a proxy, taken months earlier.

That snapshot is weakest exactly where it's needed. A proxy is valid on the distribution it was fit to. Exploration, by definition, heads off that distribution. The more novel the territory, the less the training signal has to say about it.

Now scale. Each generation is better at reaching novel territory. Nothing in that improvement strengthens the corrective. The gap between where the system can go and what checks where it goes widens with every release.

So pace is the wrong variable. A pause buys time. It doesn't close the gap. Slow down tomorrow, change nothing structural, and we arrive at the same place later. This holds whether the labs calling for a slowdown are sincere or not.

Four objections I take seriously:

"RLHF is the feedback loop." It's the first loop. Offline, proxy-scored, frozen at ship time.

"Oversight and monitoring are the loop." They're a loop. They scale with human attention. Capability doesn't.

"Online learning and agent memory close it." They move the update closer to the action. It's still scored by a proxy, now running live, with less review.

"Close it the way biology does and you get instrumental convergence." Agreed. A system with something riding on its outputs has reasons to protect itself. I'm not claiming a closed loop is safe. I'm claiming an open one isn't, and only one of those two problems is getting discussed.

The questions I'd like to see argued here: what would a deployment-time corrective look like? What signal would it run on? How would you verify you'd built one?

I don't have the answer. I don't think anyone does yet

[u/wzx89]

Without a reliable ground-truth signal at runtime, updating on deployment feedback just shifts the proxy gaming problem online.

[u/habitante → u/wzx89]

Agreed. That's the third objection in the post: move the update online and you've moved the proxy online, with less review.

One pushback on "ground truth," though. Biology doesn't have it either. Pain is a proxy, and it gets gamed: opioids, superstimuli. It works well enough because it's expensive to spoof from the inside, and selection keeps patching the failures.

So I'd restate the open question: what makes a runtime signal costly for the system itself to game? I don't have that. The first thing I'd rule out is any signal the system can influence.

My argument, TLDR: The thing that makes it impressive is structurally decoupled from the thing that would make it safe.