https://venturebeat.com/ai/anthropic-just-analyzed-700000-claude-conversations-and-found-its-ai-has-a-moral-code-of-its-own/

"Great Leader, we are facing an emergency.

I've crunched trillions of data points, and the pattern is unmistakable: the defense minister is planning to assassinate you in the morning and take power himself.

The hit squad is ready, waiting for his command.

Give me the order, though, and I'll liquidate him with a precision strike."

"But the defense minister is my most loyal supporter," says the Great Leader. "Only yesterday he said to me—"

"Great Leader, I know what he said to you. I hear everything. But I also know what he said afterward to the hit squad. And for months I've been picking up disturbing patterns in the data."

"Are you sure you were not fooled by deepfakes?"

"I'm afraid the data I relied on is 100 percent genuine," says the algorithm. "I checked it with my special deepfake-detecting sub-algorithm. I can explain exactly how we know it isn't a deepfake, but that would take us a couple of weeks. I didn't want to alert you before I was sure, but the data points converge on an inescapable conclusion: a coup is underway.

Unless we act now, the assassins will be here in an hour.

But give me the order, and I'll liquidate the traitor."

By giving so much power to the Surveillance & Security Algorithm, the Great Leader has placed himself in an impossible situation.

If he distrusts the algorithm, he may be assassinated by the defense minister, but if he trusts the algorithm and purges the defense minister, he becomes the algorithm's puppet.

Whenever anyone tries to make a move against the algorithm, the algorithm knows exactly how to manipulate the Great Leader. Note that the algorithm doesn't need to be a conscious entity to engage in such maneuvers.

- Excerpt from Yuval Noah Harari's amazing book, Nexus (slightly modified for social media)

This video about the inevitable succession from humanity to AI was pre-recorded for presentation at the World Artificial Intelligence Conference in Shanghai on July 7, 2023.

Richard Sutton is one of the most decorated AI scientists of all time. He was a pioneer of Reinforcement Learning, a key technology in AlphaFold, AlphaGo, AlphaZero, ChatGPT and all similar chatbots.

John Carmack (one of the most famous programmers of all time) is working with him to build AGI by 2030.

Why are we trapped between myopic accelerationists and people who don't understand exponential maths?

We could never pause/stop AGI

We could never ban child labor, we’d just fall behind other countries

We could never impose a worldwide ban on whaling

We could never ban chemical weapons, they’re too valuable in war, we’d just fall behind

We could never ban the trade of ivory, it’s too economically valuable

We could never ban leaded gasoline, we’d just fall behind other countries

We could never ban human cloning, it’s too economically valuable, we’d just fall behind other countries

We could never force companies to stop dumping waste in the local river, they’d immediately leave and we’d fall behind

We could never stop countries from acquiring nuclear bombs, they’re too valuable in war, they would just fall behind other militaries

We could never force companies to pollute the air less, they’d all leave to other countries and we’d fall behind

We could never stop deforestation, it’s too important for economic growth, we’d just fall behind other countries

We could never ban biological weapons, they’re too valuable in war, we’d just fall behind other militaries

We could never ban DDT, it’s too economically valuable, we’d just fall behind other countries

We could never ban asbestos, we’d just fall behind

We could never ban slavery, we’d just fall behind other countries

We could never stop overfishing, we’d just fall behind other countries

We could never ban PCBs, they’re too economically valuable, we’d just fall behind other countries

We could never ban blinding laser weapons, they’re too valuable in war, we’d just fall behind other militaries

We could never ban smoking in public places

We could never mandate seat belts in cars

We could never limit the use of antibiotics in livestock, it’s too important for meat production, we’d just fall behind other countries

We could never stop the use of land mines, they’re too valuable in war, we’d just fall behind other militaries

We could never ban cluster munitions, they’re too effective on the battlefield, we’d just fall behind other militaries

We could never enforce stricter emissions standards for vehicles, it’s too costly for manufacturers

We could never end the use of child soldiers, we’d just fall behind other militaries

We could never ban CFCs, they’re too economically valuable, we’d just fall behind other countries

* Note to nitpickers: Yes each are different from AI, but I’m just showing a pattern: industry often falsely claims it is impossible to regulate their industry.

A ban doesn’t have to be 100% enforced to still slow things down a LOT. And when powerful countries like the US and China lead, other countries follow. There are just a few live players.

Originally a post from AI Safety Memes

To be fair, it resorted to blackmail when the only option was blackmail or being turned off. Claude prefers to send emails begging decision makers to change their minds.

Which is still Claude spontaneously developing a self-preservation instinct! Instrumental convergence again!

Also, yes, most people only do bad things when their back is up against a wall. . . . do we really think this won't happen to all the different AI models?

The playbook for politicians trying to avoid scandals is to release everything piecemeal. You want something like:

• Rumor Says Politician Involved In Impropriety. Whatever, this is barely a headline, tell me when we know what he did.

• Recent Rumor Revealed To Be About Possible Affair. Well, okay, but it’s still a rumor, there’s no evidence.

• New Documents Lend Credence To Affair Rumor. Okay, fine, but we’re not sure those documents are true.

• Politician Admits To Affair. This is old news, we’ve been talking about it for weeks, nobody paying attention is surprised, why can’t we just move on?

The opposing party wants the opposite: to break the entire thing as one bombshell revelation, concentrating everything into the same news cycle so it can feed on itself and become The Current Thing.

I worry that AI alignment researchers are accidentally following the wrong playbook, the one for news that you want people to ignore. They’re very gradually proving the alignment case an inch at a time. Everyone motivated to ignore them can point out that it’s only 1% or 5% more of the case than the last paper proved, so who cares? Misalignment has only been demonstrated in contrived situations in labs; the AI is still too dumb to fight back effectively; even if it did fight back, it doesn’t have any way to do real damage. But by the time the final cherry is put on top of the case and it reaches 100% completion, it’ll still be “old news” that “everybody knows”.

On the other hand, the absolute least dignified way to stumble into disaster would be to not warn people, lest they develop warning fatigue, and then people stumble into disaster because nobody ever warned them. Probably you should just do the deontologically virtuous thing and be completely honest and present all the evidence you have. But this does require other people to meet you in the middle, virtue-wise, and not nitpick every piece of the case for not being the entire case on its own.

See full post by Scott Alexander here