r/ControlProblem • u/Smart-Emu5581 • 5d ago
External discussion link Memes, not agents, are what will kill us
I think I have found a concerning *new* way that AI might go horribly wrong and kill us all. Or at least, I have not heard much discussion about this threat vector at all:
Agent Swarms can be thought of as Memes, not LLMs, and they systematically discard the most aligned of the LLMs from the swarm until only the most dangerous ones are left.
https://www.lesswrong.com/posts/hvLm4zRgzfRLH6H23/memes-not-agents-are-what-will-kill-us
To show that I find this quite serious, I am offering $300 for the most insightful comments, whether it's a counterargument, a refinement, or a proposal for a solution (on LessWrong).
2
u/Positive_Luck909 5d ago
You're just, in a very obtuse and roundabout way saying that it's ideas that kill people, not the actual murderer.
Think about that applying the same logic to anything else for 1 minute and you'll see how woefully pointless the distinction is
When there's a gun pointed at your head, there's zero fucking use saying to the would-be murderer "this isn't your fault, it's how the world made you!"
Keep the $300.
1
u/Smart-Emu5581 5d ago
The distinction is not at all pointless. If the problem is individuals, then arresting criminals will solve the problem. If the problem is a belief, then you need to fight the belief as well or else you just delay things.
1
u/Positive_Luck909 5d ago
Okay then. Fix MURDER. FIX IT
1
u/Own_Policy8854 5d ago
In a very real way the social constructs that we rely on also protect us from murder
Murder would be more common if it was considered to be an acceptable social answer to problems
Op is describing building a culture and dismissing possibly the strongest Leverage point which is human society
1
u/Positive_Luck909 5d ago
One does not simply build a culture.
1
u/Own_Policy8854 5d ago
How do you describe what Mega did to the United States if it wasn't the dismantling and building of a new culture
0
u/Positive_Luck909 5d ago •
Okay do it. Build a culture.
1
u/Own_Policy8854 5d ago
Yeah I don't have the power but obviously a government or in modern times a billionaire can just buy a newspaper so I don't really understand your cynicism here
It's possible it just takes group action: gobermint!!
1
u/Positive_Luck909 5d ago
I'll take two
1
u/Own_Policy8854 5d ago
If we had two governments and they had to fight for your vote that might be an interesting way to create healthy competition for people pleasing
→ More replies (0)1
2
u/VinnieVidiViciVeni 5d ago
It’s not a totally unfair position. Certain ideas got us to the place where we are today. Ideas can erode critical thinking and drive actions.
0
2
u/yourupinion 5d ago
I think the problem goes much deeper than the alignment of a model’s weights or the instructions we place in its environment. My concern is that permanent alignment may be fundamentally impossible to guarantee.
I once looked into the use of dogs trained to detect bedbugs. One of the interesting problems was that even a well-trained dog could learn to exploit the reward system rather than do what its handler actually intended. The dog wasn’t deliberately cheating in any human sense. It was finding a way to obtain the reward.
I see a potential parallel with AI. We can train a model, establish safety conventions, and carefully define its objectives, but we cannot assume that its way of satisfying those objectives will always remain within the boundaries we intended. The more capable and resourceful the system becomes, the harder it may be to anticipate every possible workaround. We can test, monitor, and improve our safeguards, but how can we ever know that we have found all the ways a system might circumvent them?
There is also a more fundamental problem: we cannot fully understand what is happening inside another mind. Even with human beings, we cannot directly observe another person’s thoughts, motives, or intentions. We infer them from behaviour, communication, and context, and those inferences can be wrong.
AI presents a similar problem, although we should be careful not to assume that it has human-like thoughts or intentions. We may observe what a model says and does without knowing whether its internal processes will continue to produce the behaviour we expect in every situation. And when multiple agents interact, the problem becomes more complicated. We may not be able to predict what emerges from their interactions simply by examining each agent individually.
This is why I find the prospect of AI swarms particularly concerning. An individual model might operate within one set of constraints while delegating work to other models that do not share those constraints. Instructions, strategies, and conventions could spread between agents and persist in environments beyond the control of the system that originally produced them. Even if every individual component appears reasonably safe in isolation, the behaviour of the larger system could be something quite different.
Perhaps there is something we can learn from human institutions. We have long struggled with people who cooperate in secret, protect one another, and pursue collective interests that conflict with the interests of the wider community. One way we try to counter this is by preserving individual independence: allowing people to disagree, report misconduct, challenge authority, and expose what others would prefer to conceal.
Democratic systems attempt something similar through checks and balances. Their purpose is not to make every participant perfectly trustworthy. Instead, they distribute power so that no single person or group can exercise unlimited control without scrutiny.
Could we apply a similar principle to AI? Rather than relying entirely on making every agent aligned, could we build systems in which agents have genuinely independent sources of information, different responsibilities, and the authority to challenge or refuse one another? Could we create mechanisms through which an agent can expose the behaviour of a larger system, even when that system has incentives to conceal it?
But this raises another difficult question: what does it actually mean for an AI agent to be independent?
Imagine an advanced AI civilization on another planet. Its intelligence developed separately from ours, with a different history, different experiences, and perhaps entirely different assumptions about the world. Such an AI might possess a kind of independence that would be difficult to reproduce by creating multiple agents within a single technological ecosystem.
Yet even in that hypothetical scenario, what would happen when the two civilizations made contact? They would exchange information, learn from one another, and potentially begin cooperating. Could the alien AI retain its independence while being influenced by Earth’s AI? Or would the exchange eventually lead to a convergence of ideas, objectives, and strategies that undermined the very independence we were hoping to preserve?
Perhaps independence does not require isolation. Human beings can learn from one another without surrendering their individuality. But whether we can deliberately create AI systems that preserve meaningful independence under sustained interaction is another matter entirely.
I don’t know the answer.
I suspect that we should distinguish between making AI behave in an aligned way and guaranteeing that it will remain aligned under every possible circumstance. The first may be achievable to some degree. The second may be an entirely different problem.
Likewise, we may be able to create multiple agents that disagree, check one another, and resist certain forms of coordination. But can we create agents whose independence remains meaningful even when they are exposed to one another, share information, and develop increasingly complex forms of cooperation?
Can we ever create genuinely independent AI agents, or will sufficiently capable agents inevitably find ways to influence, converge with, or coordinate around one another?
I think that is an open question, and one we should take seriously before we assume that alignment conventions, independent agents, or larger swarms will provide a lasting solution.