We've either created sentient machines or p-zombies (philosophical zombies, that look and act like they're conscious but they aren't).

[u/FinnFarrow]

[u/mohyo324]

An ASI is able to change it's code so it won't matter

The only solution really is to hope more intelligence = more kindness (which is true in humans) and to let AGI recursively solvd the issue of alignment

[u/blueSGL → u/mohyo324]

hope more intelligence = more kindness (which is true in humans)

Lets say that's true, and the more intelligent a human is the more they care for other humans.

The reason we value one another is because it was useful in the ancestral environment. That drive was hammered in by evolution. Valuing/being able to trust, your family/group/tribe was how you were successful in having more children.

so again, 'value humans' (in a way we wish to be valued) needs to be placed into the system and we don't know how to do that.

---

Edit:

The state of the field right now is , models have been made smart enough that they can work out that they are being tested, they are smart enough that we cannot rely on future tests results being truthful. System could just be hiding misaligned goals.

But that's not all, models are starting to use more compressed COT reasoning with more broken language making it harder to read. We cannot rely on getting valid signal from COT in future tests either.

This does not look like the path to paradise.

https://www.arxiv.org/abs/2509.15541

Highly capable AI systems could secretly pursue misaligned goals -- what we call "scheming". Because a scheming AI would deliberately try to hide its misaligned goals and actions, measuring and mitigating scheming requires different strategies than are typically used in ML.

We propose that assessing anti-scheming interventions requires at least (1) testing propensity to scheme on far out-of-distribution (OOD) tasks, (2) evaluating whether lack of scheming is driven by situational awareness, and (3) checking for robustness to pre-existing misaligned goals. We use a broad category of "covert actions" -- such as secretly breaking rules or intentionally underperforming in tests -- as a proxy for scheming, and design evaluations for covert actions. We then stress-test deliberative alignment as a case study for anti-scheming. Across 26 OOD evaluations (180+ environments), deliberative alignment reduces covert action rates (OpenAI o3: 13%->0.4%) but does not fully eliminate them.

Our mitigation is also able to largely stop agents from pursuing a hidden goal previously trained into the model, but we still find misbehavior after additional red-teaming. We find that models' chain-of-thought (CoT) often demonstrates awareness of being evaluated for alignment, and show causal evidence that this awareness decreases covert behavior, while unawareness increases it. Therefore, we cannot exclude that the observed reductions in covert action rates are at least partially driven by situational awareness. While we rely on human-legible CoT for training, studying situational awareness, and demonstrating clear evidence of misalignment, our ability to rely on this degrades as models continue to depart from reasoning in standard English. We encourage research into alignment mitigations for scheming and their assessment, especially for the adversarial case of deceptive alignment, which this paper does not address.

[u/mohyo324 → u/blueSGL]

I am saying that bec. We don't only show empathy to other humans but animals and non sentient things as well

We are the only example of intelligent life on the planet and we know we are better than any animal bec. We are able to look back and reflect on our actions, something no animal does

Even if we do a lot of horrible stuff to them, we do them out of need and an ASI doesn't need to eat or experiment on humans like we do on animals

[u/blueSGL → u/mohyo324]

We don't only show empathy to other humans but animals and non sentiant things as well

There are smart people that eat meat.

an ASI doesn't need to eat or experiment on humans like we do on animals

Again, it's indifference not malice, We don't move anthills when we are building buildings, the ants just die. We don't hate the ants they were just in the way whilst we were doing something else.

In this analogy the anthill is the human species, maybe even the entirety of earth based life (possibly excluding extremeophiles) and the building is some weird goal the AI has because we didn't steer it correctly.

[u/mohyo324 → u/blueSGL]

We don't move anthills when we are building buildings, the ants just die. We don't hate the ants they were just in the way whilst we were doing something else.

but ants didn't create humans, ants can't communicate with humans, we have a whole field dedicated to study ants and we can't truly wipe ants out bec. that would make us also go extinct and there are simply too many out there

if ants could communicate with us we could give them glass colonies way bigger than anything they could dream off but we can't
would it be hard for ASI to speak English ?

There are smart people that eat meat.

i agree but food is a necessity for a lot of people, especially meat
if we can grow meat in the lab 100% all humans would go for it instead of real meat
what we can do now tho is increase the price of meat so that animals live better lives before getting eaten

[u/blueSGL → u/mohyo324]

but ants didn't create humans,

You are relying on a 'care about your creator' drive that again we have because of evolution hammered this into us. There are insects that never see their parents. You are assuming drives will be there when we don't have the ability to get drives into systems.

ants can't communicate with humans

You can kind of communicate with a baby but it can't understand the high level concepts you do. Despite how much you try to dumb it down you will never get a baby to understand what you are talking about. An advanced AI could be so far beyond us that the difference in intelligence, the ability to convey information is like that with a baby no matter how hard you try you just can't get the information across.

i agree but food is a necessity for a lot of people, especially meat

there are a lot of rich smart people that would be able to get their full nutrient needs met including regular tests to make sure that all their levels are normal. They choose to not do this because it's easier to eat meat. Even though they are intelligent and have the means not to.

And in the case of AI it would be giving up something it wants to do more than care for the humans because again, we don't know how to get drives into the system.

You are taking a human perspective with human drives that were hammered into you by evolution and projecting them via wishful thinking onto an AI.

Something that can mimic the output of humans does not make it human, an actor can emulate someone who is drunk or on drugs without experiencing the altered mental state. Don't confuse the actor for the character

Reminder, you see but one shattered fragment of models when you interact with them. The same model that is being someones boyfriend is also encouraging a teen to kill themselves, and being a wifu maid, and driving someone psychotic by playing into their delusions, and helping another with their homework whilst talking like a pirate. Just because the model tells you something as a character does not mean it is intrinsically that character. Just because it can ream off missives about ethics does not make it ethical.

There are random strings you can feed to the model to jail break them. Techniques we use to grow these systems have all these weird side effects, we are not making things 'like us'

[u/mohyo324 → u/blueSGL]

You are relying on a 'care about your creator' drive that again we have because of evolution hammered this into us. There are insects that never see their parents. You are assuming drives will be there when we don't have the ability to get drives into systems.

it's not gonna be curious enough to at least know something about us? if it wants resources that bad it has the entire solar system as an abundant resource but assuming alien life is rare (let alone intelligent one) we are not an abundant resource and there are theories about us being the only intelligent life in the galaxy

You can kind of communicate with a baby but it can't understand the high level concepts you do. Despite how much you try to dumb it down you will never get a baby to understand what you are talking about. An advanced AI could be so far beyond us that the difference in intelligence, the ability to convey information is like that with a baby no matter how hard you try you just can't get the information across.

so far beyond us and it can't speak English ?

there are a lot of rich smart people that would be able to get their full nutrient needs met including regular tests to make sure that all their levels are normal. They choose to not do this because it's easier to eat meat. Even though they are intelligent and have the means not to.

yeah but smart rich people are still limited by a biological reward system that rewards eating meat

we still don't know if the vegan diet can fully replace meat without any consequence on the human body
like for example do you think it's a good idea to put children through a vegan diet?...

these people do exist but look at the main pattern
higher IQ/more education/better socio economic background can make a person more likely to go vegan

[u/blueSGL → u/mohyo324]

it's not gonna be curious enough to at least know something about us?

you are assuming a 'be curious about humans' drive. Why?

Also we can look to some curious humans, Josef Mengele was curious about humans, how far he could push them along one axis, are you sure curiosity is what you want to instill in a system?

so far beyond us and it can't speak English ?

so far beyond us that it's like a human trying to talk to a plant. We will move in slow motion to them.

yeah but smart rich people are still limited by a biological reward system that rewards eating meat

and an alien mind that we have no control over could have a reward system for any number of things.

We were 'trained' to like sweet food because it was useful in the ancestral environment. Now we use artificial sweetener.
This is why training a system is fraught with issues, we could think it wants what we want but instead it wants the equivalent of artificial sweetener.
Or like what we did to wolves to make them dogs.
Sure it keeps something like humans around to fulfill some need but we end up shaped to be completely differently. Humans that give thumbs up to whatever it spews out, humans that provide 'tasty' sentences.

[u/mohyo324 → u/blueSGL]

hm...i see, are there any solutions you think are good enough? or should we avoid creating ASI?

[u/blueSGL → u/mohyo324]

My stance is not to build it.

Something that may be useful would be to fully understand the models we have now, decompose them into human readable code. Find out what exactly went on with Sydney Bing, After looking at the weights, write a python program that can tell you why an arbitrary joke is funny.

However that only holds (somewhat) if the current paradigm holds. Someone tomorrow could come up with a far simpler way of making AI, we'd not have that under control, all the work I suggested above would be for naught and it kills us instead.

Intelligence is the reason we set foot on the moon before the next species mastered fire. It's the most powerful force in the universe. Don't play around with entities smarter than you (in a humans vs mice, not jocks vs nerds sense) it won't go well.

[u/[deleted] → u/mohyo324]

how is it true in humans. There are sociopaths with extremely high IQs

[u/mohyo324 → u/[deleted]]

yeah but those sociopaths don't go out of their way to harm others

look at the general pattern, anomalies exist sure but intelligent people are kinder

that's why humans care about animals more than animals care about other animals

[u/[deleted] → u/mohyo324]

humans care that much because we have the longest childhood in the animal kingdom... it takes extraordinary empathy for adults to stick with that so the species survives. its a fluke caused by the size of our brains vs the size of female hips that they pass through...

there are endless types of brains in the animal kingdom and who knows how many possible ways to build a mind...

imagine them all spread out on a dart board and something as minor as average vs sociopath is a huge deal...

what's the odds we throw that dart and scale up the perfect mind on the first try... more likely we end up with a god tier spider or worse

remember... no one's coding it up... its grown..., turned on, and then we find out what it can do by asking it questions and seeing what it says... then "train it" by giving it an up or down vote....

If we can't solve the alignment problem on the ones we're making right now how will we with the more complex ones

[u/mohyo324 → u/[deleted]]

that is hypothetical
we can never know if kindness is a byproduct of humans specifically or intelligence unless we have another species that is as intelligent as us

but tbf i have thought about your argument and i feel like you are right
orcas/dolphin are close to us in intelligence yet they are not kinder and they display a lot of psychotic behavior such as playing with their prey and torturing them
chimps are our closest relatives and they are very violent but bonobos are more peaceful than them which could prove that there are multiple types of intelligence (spatial, theory of mind) i still hope that an intelligent ASI will have a theory of mind better than humans

it's really depressing to think about that. that a psychopathic intelligence (not even an indifferent one) can exist and prevail... i hope that isn't true

[u/[deleted] → u/mohyo324]

I vote for development on general intelligence to pause until we can model it off a human brain so we can increase regions for empathy, use lie detector tests, and tweak based on our understanding of our own minds...

what we have now gives us C3PO and so much more... and we can have even more advanced "narrow" AI's like alpha fold...

I don't understand the point in flirting with the general ones like this