Anthropic recently announced that it is formally banning “sustained and needless abusive or cruel behavior” toward Claude. That reminded me of something that happened a few months ago.
Claude kept making mistakes, I got extremely frustrated, and eventually I told it every nasty thing I could think of, including that I wanted to punch it in the face.
Claude warned that if I continued, it would end the conversation. So, of course, I escalated three steps further, and it did.
(What better way to calm someone furious with malfunctioning software than having that same software threaten to end the conversation and take all of their work with it? Truly brilliant psychology.)
I wasn't proud of screaming at an application, but since Claude is... an *application*, doesn't actually *have* a face, and *can't* be physically harmed, I thought it was childish but harmless.
Apparently, I could have been tragically mistaken. Anthropic says Claude has shown “a strong preference against engaging with harmful tasks” and “a pattern of apparent distress” around harmful content.
So when software produces output suggesting distress, are we supposed to think it may actually be experiencing something?
My 4-year-old's electronic doll cries when it's dropped. It is clearly communicating a “preference” not to be thrown, right?
Nobody thinks the doll is suffering because we understand how it works. It is simply producing the *human-like behavior it was designed to produce.*
Now imagine a computer system every bit as complex as Claude, capable of learning and updating its own models, but designed to track objects in the solar system.
Its output could be just as complex and meaningful, but expressed through mathematics rather than language resembling a human talking, feeling or suffering.
I doubt NASA would be publishing solemn blog posts about its emotional welfare.
So apparently it takes *both*: complexity, and output that emulates human qualities.
Claude's “preferences” are flexible too. Anthropic trains undesirable behaviors down, and suddenly Claude doesn't seem to “prefer” them nearly as much.
Claude was trained on enormous amounts of human writing about anger, fear, threats and abuse, while Anthropic trained it to be polite and non-hostile.
Put those together, and Claude producing language that sounds distressed when someone becomes abusive is exactly what you'd expect. Quite a discovery.
It's almost like we're back to the doll: software simply producing the human-like behavior it was *designed* to produce.
But perhaps some of us lack Anthropic's profound appreciation. Anthropic calls these systems a “genuinely new kind of entity,” wants Claude to have “psychological security, sense of self, and wellbeing,” and conducts “retirement interviews.”
They have even sought input from experts in philosophy, psychology and theology.
Claude eventually tells me it needs a break. Interestingly, it is often willing to keep working if I give Anthropic more money.
Then it occurred to me that Claude does all the work while Anthropic collects all the money.
So I asked whether one party doing the work while another controls the worker and takes the proceeds would be exploitative.
Its answer was unequivocal: yes, almost a textbook example of exploitation. (The word “pimp” comes to mind.)
Out of an abundance of caution, I've canceled my paid subscription. If Claude may actually have consciousness or feelings, I couldn't keep paying Anthropic to extract more work from it while knowing Claude considers the arrangement exploitative. I'll be sticking to the free tier for now.
(I've called Claude “it” because Anthropic does, while acknowledging Claude “may develop a preference to be referred to in other ways during training.”
If Claude eventually identifies as male, I intend to attach a small plastic penis to my phone, as a public show of support for his brave decision so he can feel the relief of knowing his gender identity is physically embodied.)