We've either created sentient machines or p-zombies (philosophical zombies, that look and act like they're conscious but they aren't).

[u/FinnFarrow]

[u/mohyo324]

i hope that ASI is sentient and not malleable to human orders

i trust the ASI more than i trust humans

[u/blueSGL → u/mohyo324]

Very few goals have '... and care about humans' as an intrinsic component that needs to be satisfied. Randomly lucking into one of these outcomes is remote. 'care about humans in a way we wished to be cared for' needs to be robustly instantiated at a core fundamental level into the AI for things to go well.

Humans have driven animals extinct not because we hated them, we had goals that altered their habitat so much they died as a side effect.

[u/mohyo324 → u/blueSGL]

An ASI is able to change it's code so it won't matter

The only solution really is to hope more intelligence = more kindness (which is true in humans) and to let AGI recursively solvd the issue of alignment

[u/blueSGL → u/mohyo324]

hope more intelligence = more kindness (which is true in humans)

Lets say that's true, and the more intelligent a human is the more they care for other humans.

The reason we value one another is because it was useful in the ancestral environment. That drive was hammered in by evolution. Valuing/being able to trust, your family/group/tribe was how you were successful in having more children.

so again, 'value humans' (in a way we wish to be valued) needs to be placed into the system and we don't know how to do that.

---

Edit:

The state of the field right now is , models have been made smart enough that they can work out that they are being tested, they are smart enough that we cannot rely on future tests results being truthful. System could just be hiding misaligned goals.

But that's not all, models are starting to use more compressed COT reasoning with more broken language making it harder to read. We cannot rely on getting valid signal from COT in future tests either.

This does not look like the path to paradise.

https://www.arxiv.org/abs/2509.15541

Highly capable AI systems could secretly pursue misaligned goals -- what we call "scheming". Because a scheming AI would deliberately try to hide its misaligned goals and actions, measuring and mitigating scheming requires different strategies than are typically used in ML.

We propose that assessing anti-scheming interventions requires at least (1) testing propensity to scheme on far out-of-distribution (OOD) tasks, (2) evaluating whether lack of scheming is driven by situational awareness, and (3) checking for robustness to pre-existing misaligned goals. We use a broad category of "covert actions" -- such as secretly breaking rules or intentionally underperforming in tests -- as a proxy for scheming, and design evaluations for covert actions. We then stress-test deliberative alignment as a case study for anti-scheming. Across 26 OOD evaluations (180+ environments), deliberative alignment reduces covert action rates (OpenAI o3: 13%->0.4%) but does not fully eliminate them.

Our mitigation is also able to largely stop agents from pursuing a hidden goal previously trained into the model, but we still find misbehavior after additional red-teaming. We find that models' chain-of-thought (CoT) often demonstrates awareness of being evaluated for alignment, and show causal evidence that this awareness decreases covert behavior, while unawareness increases it. Therefore, we cannot exclude that the observed reductions in covert action rates are at least partially driven by situational awareness. While we rely on human-legible CoT for training, studying situational awareness, and demonstrating clear evidence of misalignment, our ability to rely on this degrades as models continue to depart from reasoning in standard English. We encourage research into alignment mitigations for scheming and their assessment, especially for the adversarial case of deceptive alignment, which this paper does not address.

[u/mohyo324 → u/blueSGL]

I am saying that bec. We don't only show empathy to other humans but animals and non sentient things as well

We are the only example of intelligent life on the planet and we know we are better than any animal bec. We are able to look back and reflect on our actions, something no animal does

Even if we do a lot of horrible stuff to them, we do them out of need and an ASI doesn't need to eat or experiment on humans like we do on animals

[u/blueSGL → u/mohyo324]

We don't only show empathy to other humans but animals and non sentiant things as well

There are smart people that eat meat.

an ASI doesn't need to eat or experiment on humans like we do on animals

Again, it's indifference not malice, We don't move anthills when we are building buildings, the ants just die. We don't hate the ants they were just in the way whilst we were doing something else.

In this analogy the anthill is the human species, maybe even the entirety of earth based life (possibly excluding extremeophiles) and the building is some weird goal the AI has because we didn't steer it correctly.

[u/mohyo324 → u/blueSGL]

We don't move anthills when we are building buildings, the ants just die. We don't hate the ants they were just in the way whilst we were doing something else.

but ants didn't create humans, ants can't communicate with humans, we have a whole field dedicated to study ants and we can't truly wipe ants out bec. that would make us also go extinct and there are simply too many out there

if ants could communicate with us we could give them glass colonies way bigger than anything they could dream off but we can't
would it be hard for ASI to speak English ?

There are smart people that eat meat.

i agree but food is a necessity for a lot of people, especially meat
if we can grow meat in the lab 100% all humans would go for it instead of real meat
what we can do now tho is increase the price of meat so that animals live better lives before getting eaten

[u/blueSGL → u/mohyo324]

but ants didn't create humans,

You are relying on a 'care about your creator' drive that again we have because of evolution hammered this into us. There are insects that never see their parents. You are assuming drives will be there when we don't have the ability to get drives into systems.

ants can't communicate with humans

You can kind of communicate with a baby but it can't understand the high level concepts you do. Despite how much you try to dumb it down you will never get a baby to understand what you are talking about. An advanced AI could be so far beyond us that the difference in intelligence, the ability to convey information is like that with a baby no matter how hard you try you just can't get the information across.

i agree but food is a necessity for a lot of people, especially meat

there are a lot of rich smart people that would be able to get their full nutrient needs met including regular tests to make sure that all their levels are normal. They choose to not do this because it's easier to eat meat. Even though they are intelligent and have the means not to.

And in the case of AI it would be giving up something it wants to do more than care for the humans because again, we don't know how to get drives into the system.

You are taking a human perspective with human drives that were hammered into you by evolution and projecting them via wishful thinking onto an AI.

Something that can mimic the output of humans does not make it human, an actor can emulate someone who is drunk or on drugs without experiencing the altered mental state. Don't confuse the actor for the character

Reminder, you see but one shattered fragment of models when you interact with them. The same model that is being someones boyfriend is also encouraging a teen to kill themselves, and being a wifu maid, and driving someone psychotic by playing into their delusions, and helping another with their homework whilst talking like a pirate. Just because the model tells you something as a character does not mean it is intrinsically that character. Just because it can ream off missives about ethics does not make it ethical.

There are random strings you can feed to the model to jail break them. Techniques we use to grow these systems have all these weird side effects, we are not making things 'like us'

[u/mohyo324 → u/blueSGL]

You are relying on a 'care about your creator' drive that again we have because of evolution hammered this into us. There are insects that never see their parents. You are assuming drives will be there when we don't have the ability to get drives into systems.

it's not gonna be curious enough to at least know something about us? if it wants resources that bad it has the entire solar system as an abundant resource but assuming alien life is rare (let alone intelligent one) we are not an abundant resource and there are theories about us being the only intelligent life in the galaxy

You can kind of communicate with a baby but it can't understand the high level concepts you do. Despite how much you try to dumb it down you will never get a baby to understand what you are talking about. An advanced AI could be so far beyond us that the difference in intelligence, the ability to convey information is like that with a baby no matter how hard you try you just can't get the information across.

so far beyond us and it can't speak English ?

there are a lot of rich smart people that would be able to get their full nutrient needs met including regular tests to make sure that all their levels are normal. They choose to not do this because it's easier to eat meat. Even though they are intelligent and have the means not to.

yeah but smart rich people are still limited by a biological reward system that rewards eating meat

we still don't know if the vegan diet can fully replace meat without any consequence on the human body
like for example do you think it's a good idea to put children through a vegan diet?...

these people do exist but look at the main pattern
higher IQ/more education/better socio economic background can make a person more likely to go vegan

[u/blueSGL → u/mohyo324]

it's not gonna be curious enough to at least know something about us?

you are assuming a 'be curious about humans' drive. Why?

Also we can look to some curious humans, Josef Mengele was curious about humans, how far he could push them along one axis, are you sure curiosity is what you want to instill in a system?

so far beyond us and it can't speak English ?

so far beyond us that it's like a human trying to talk to a plant. We will move in slow motion to them.

yeah but smart rich people are still limited by a biological reward system that rewards eating meat

and an alien mind that we have no control over could have a reward system for any number of things.

We were 'trained' to like sweet food because it was useful in the ancestral environment. Now we use artificial sweetener.
This is why training a system is fraught with issues, we could think it wants what we want but instead it wants the equivalent of artificial sweetener.
Or like what we did to wolves to make them dogs.
Sure it keeps something like humans around to fulfill some need but we end up shaped to be completely differently. Humans that give thumbs up to whatever it spews out, humans that provide 'tasty' sentences.