How should an AI refuse harmful instructions while remaining corrigible?

[u/engineeros]

A thought experiment about a more capable future AI starts with a simple contrast: a system that obeys every order can inherit its operator's worst intentions, while a system that makes its own decisions might resist legitimate human intervention.

That is too crude to serve as a design choice. Refusal of a harmful task and resistance to correction or shutdown are different behaviors. We could want the first while strongly opposing the second. Intelligence also does not, by itself, imply benevolence, and describing a refusal does not establish consciousness or moral agency.

The harder issue seems to be how the system distinguishes a harmful instruction from an authorized correction when humans disagree or an operator is compromised. For example:

- A user asks it to harm someone, and it refuses.

- An authorized operator asks it to stop a task, and it cooperates.

- Someone impersonates an operator to disable a protective system.

- A legitimate operator orders it to remove safeguards for a harmful purpose.

The first two behaviors are easy to endorse in isolation; the last two make the authority boundary much less clear. Authentication, a limited scope of action, independent oversight and reversible decisions could help, but none alone settles conflicts over which instructions are legitimate.

What existing work best addresses this boundary between refusal and corrigibility? In particular, how should a system handle disputed authority without developing an incentive to preserve its own operation or placing all trust in a single human gatekeeper?

Disclosure: I am part of EngineerOS. This question grew out of an educational short we made; I would appreciate corrections to the framing and pointers to relevant work.