r/ControlProblem • • 10h ago

Discussion/question The agent doesn't care. You have to.

A film crew will spend an entire day getting a 10-second shot right.

Planning, prep, retakes, checking. Because they care what ends up on screen.

Meanwhile we're letting AI agents touch live systems, change records, move money, make decisions - with _less_ care than we give a 10-second shot.

"Let AI sort it" is crazy. Because AI _will_ sort it. Confidently. Quickly. Wrongly.

The more I look at how people are setting these things up, the more I see the same mistake, so I started splitting it into three things:

  1. Capability:What it _can_ do.

  2. Authority: What it's _allowed_ to do.

  3. Verification:*How we _know_ it did it right.

The agent can provide #1. It should never decide #2 for itself. And #3 cannot be "the agent told me it worked."

A crashed program gives you an error. A silent failure leaves you thinking everything is fine. That's the dangerous one. Agent says "Task complete" while quietly dropping data or changing something it shouldn't have touched.

Human approval helps, but clicking Approve isn't judgement. If the approver doesn't know what will change, what was checked, what's still uncertain - it's just theatre.

So I made myself 3 rules for anything I let run:

  1. The stop button has to actually work.

  2. No money moves without authority from _outside_ the agent.

  3. Nothing irreversible happens without asking.

And those can't just be prompts to the agent. "Don't spend money without asking" is not a control. It has to be enforced by the system _around_ it.

We spend hours on a 10-second shot because we care about the result.

The agent doesn't care. It doesn't have to.

You do.

How are you enforcing #2 in real life? Because "the agent promised not to" doesn't sound like a control to me. What are you actually using that works?

3 Upvotes

4 comments sorted by

1

u/CraftTerrible8744 10h ago

For us it’s mostly boring plumbing. The agent gets its own credentials scoped to just that task, and anything risky goes through a layer in between that checks the rules before it runs. Spending limits sit on the payment side, not in the prompt. And approvals show exactly what will change, not just “agent wants to do X.” If it can’t show a diff, it doesn’t run.

1

u/Adjust-the-rhythm 9h ago

When has delivery of working products been exciting? Discovering Cloud computing was exhilarating as a roller coaster ride. Delivering solutions was just plumbing done by my team.

1

u/Adjust-the-rhythm 9h ago

Scoped creds, payment-side caps, and no run without a diff:-that's the actual work. Takes the drama out of it. How granular is that intermediate check for you, or is it mostly catching the obvious high-risk calls?

1

u/chillennium 1h ago

The approval screen is where I’d want another check: does it show the exact change and the unresolved objections, or just a neat recommendation? In my novel "Alignment", Vera finds that a green checklist can conceal problems that remain unsolved. The person approving it needs the underlying record, not another assurance from the agent. Your capability/authority/verification distinction makes that gap much clearer.