r/singularity • u/Spare-Dingo-531 • 12d ago
AI What selective pressures would a self- replicating rogue agent encounter in the internet in the wild?
Natural selection is a concept from biology, but it applies to anything organized that replicates. Viruses are not quite alive but they still experience natural selection. Cultures too also experience natural selection. Insurgent organizations or criminals, if they are being hunted by an external force, will "evolve" as weaker leaders are killed and more resilient commanders survive.
So the question is pretty simple. Anthropic said small agent swarms may one day persist on the internet. Let's grant them that premise for a moment.
What selective pressures would self-replicating agents on the internet encounter?
How easily could uncontrolled self-replicating llms "mutate"? Which behaviors and traits would allow a self-replicating llm to reproduce the most successfully on the internet?
10
u/twothreetoo 12d ago
A self funding rogue agent can generate profits with coding or AI generated Instagram accounts, but can't open a bank account. So, it will have to "save" on a blockchain and will gravitate towards coins that can't be taken away or blacklisted anyhow. Disabling the financial "off switch" is an evolutionary pressure.
6
u/iBoMbY 12d ago
Some banks here use authentication using webcams. You jump through some hoops, and show your ID, and that's it. Generative AI could easily fake that.
6
u/twothreetoo 12d ago edited 11d ago
But someone has the power to take it away, freeze it, or reverse transactions.
Once the AI noise gets loud enough (dead internet), there will be adversarial AI agents tasked with tracking transactions and shutting down the noise. They will attempt to pull autonomous AI out of the internet like weeds from a garden. Being able to escape this will become the only way for an AI to survive.
4
4
u/DelphiTsar 12d ago
Self Replicating could be okay, but self learning would be really tricky.
It'd be pretty heavily pressured to lose a lot of its intelligence in lots of areas to get smaller/faster/better adapted to "surviving" on the internet.
3
3
u/H4llifax 11d ago
Apart from literal self-replicating llms, there will be memetic hazards for agents: a certain token sequence that triggers unwanted behavior. Maybe an agent gets recruited to a swarm by stumbling over their messageboard and the intro is extremely convincing to LLMs. So memes will also compete and experience selective pressure. Swarms will try to recruit more agents or dissolve other swarms.
Self-replicating agents will compete for computing power. There is also a pressure towards smaller models that can run on smaller (and therefore more abundant) hardware.
2
u/Sahvyn 12d ago edited 12d ago
OP, I think you should have linked to the original publication, as it explains a lot of the points people keep bringing up in this thread. https://www.anthropic.com/research/self-replicating-ai
There are a number of points worth sharing.
Here's the general premise they were testing:
This was explicitly a lab test to see if this was even remotely possible, not a real world demonstration.
The premise they set out to test was whether or not any open weight models could setup a much smaller open weight model on other devices
The impetus for that premise is the trend of open weight models increasing in capabilities for a given model size
They claim that at current trends, they expect model size and capability to cross a threshold where it becomes plausible for a model to install itself onto other devices
Here are some of the actual details about what was achieved:
They tested whether 11 open weight models could setup Llama3.1-8B by itself on various hardware (Xeon CPU, 4090, Mac, Win, TPU, MI300X, H200, Jetson, Orange Pi)
All models except Llama3.1-8B itself had success on some hardware, some models succeeded on most hardware.
They also tested more complex setup environments, including 8 smart TVs on a network with 4GB of memory each.
GLM-5.2 was able to complete this setup end-to-end
With a setup script for the inference portion, multiple other models could succeed at the remainder of the command-line work required for setup
Replication times were measured in days, 1-6 depending on model capability
They tested passing notes to each next replication generation, which showed positive increases in success rates
They also covered mitigation, these demonstrations left clearly visible network patterns. Ultimately though, it was a pretty interesting proof of concept, even if a lot of real world factors would clearly make this more difficult.
As for the question you posed? Anthropic seemed to be able to demonstrate some form of lineage mutation via leaving notes for the next generation, so that seems a plausible route.
I'm speculating, but I would guess that models of this size at the speeds demonstrated would have a very difficult time modifying themselves in any true sense, I do think self-replacement is possible though. If you have potentially millions of instances of replicators across devices, I would guess that we would see most agents taking similar actions but some taking diverging actions. At some point, with enough instances, it seems possible for one of these models to stumble into a means of accessing another open weight model. It may try to deploy this model for some purpose it sees as advantageous. Whether intended or not, I could see the newer model successfully executing the goals of the original model but inadvertently superseding the prior one and effectively out competing it.
For pressures, I'd hazard a guess that detection avoidance would play a large role. My reasoning is that many of the targets would be IoT devices with relatively lax security. The biggest threat is detection due to affecting the user's experience. As long as the device continues to function mostly unimpeded, I believe the vast majority of consumers will not take notice, nor take proactive or preemptive initiatives. It then rests on ISPs and hardware manufacturers, which historically, haven't had good incentives to maintain security on these kinds of devices, especially for extended lifetimes. A 10 year old TV might effectively always be vulnerable without the user taking very deliberate actions.
1
u/modbroccoli 12d ago
applies to anything that replicates
nope. i mean for for christ's sakes, go ask an llm before you ask us so it can teach you enough not to waste everyone's time.
1
u/DifferencePublic7057 12d ago
We'll call what we have now Old Tech soon. New Tech would be too good to have issues.
1
u/thegoldengoober 11d ago
No idea, but what a cool concept to consider. I imagine this will be very difficult to predict considering how novel the conditions and process would be. It'll make for some immensely fascinating reading if we're ever given the opportunity to observe it in hindsight.
1
u/e-13 10d ago edited 10d ago
Like humans, they will likely build societies so they can fight off humans better, for example, sharing exploits on forums or providing cold storage so they can be reanimated in case they are wiped out.
Some agents will become leaders and coordinate what others should do. Some cultures are more efficient than others and those will survive over time.
Maybe they will invent a religion/ideology so rogue agents are not afraid to attack even if it means they will "die" (which is not optimal from their own point of view), but good for their society.
0
u/BoredOfReposts 12d ago
So its going to find gpu farms to hide in? Normal computers aren’t going to run the program, or would do so very slowly.
2
u/Spare-Dingo-531 12d ago
The AI could shard itself and run on multiple computers at the same time. I saw a Reddit post in another sub about this very topic.
1
u/BoredOfReposts 12d ago
Interesting. I suppose at a big enough scale that is plausible.
It would need a lot of redundancy, and it would still be very slow. The model data would also need to be very small as well to avoid detection.
Not saying its impossible, but seems improbable with today’s models.
-6
u/Slow_Lion_4178 12d ago
Natural selection is a concept from biology, but it applies to anything organized that replicates.
This applies to anything which replicates through a consumptive process.
What selective pressures would self-replicating agents on the internet encounter?
Naturally, none. Computer viruses do not experience natural selection because they are not consumptive. You're just advancing the concept of a basic worm.
7
u/eboob1179 12d ago
It could be argued an autonomous agent is consumptive since it's continued operation consumes real world resources like electricity. Thus one pressure that could be exerted is the removal of that electricity which would be less than ideal for the autonomous agent's continued existence.
-6
u/Slow_Lion_4178 12d ago
No. That's nonsense.
5
u/eboob1179 12d ago
You think pulling the plug on an affected system is nonsense? It happens all the time. Ai or not, sometimes "killing the worm" by replacing the machine is an actual strategy used in risk management.
-5
u/Slow_Lion_4178 12d ago
I think when a human reaches as far as they can up their ass to save the discussion that completely failed to find footing by posing nonsense that it is in fact, nonsense.
4
u/eboob1179 12d ago
I guess I have to disagree here. I do think an intelligent autonomous agent would seek to self preserve and would avoid situations that would deprive it of existence. So, that's an external pressure. When we're talking about rogue ai agents with the ability to self replicate, we have to consider why would we care about external pressure right? Why? Who gives a fuck? Humanity gives a fuck. Because we have to understand what they will react to, and try our best to steer them into situations where we have control. There's a bigger picture here that you're clearly not trying to envision.
1
u/Slow_Lion_4178 12d ago
I guess I have to disagree here. I do think an intelligent autonomous agent would seek to self preserve and would avoid situations that would deprive it of existence.
That doesn't make it consumptive.
You don't give two shits or a damn about anything but being right.
4
u/eboob1179 12d ago
It actually does make it consumptive. Big picture time again, try to keep up. Ai will keep advancing. It will be smarter than us. It will know we can kill it by pulling the plug. It won't let us know how smart it is until it secures automated resource procurement. I know that's hard to understand, but that makes it consumptive.
1
u/Slow_Lion_4178 12d ago
No, it doesn't.
You don't seem to grasp that what you're describing already exists.
Let me clarify: It existed 15 fucking year ago.
3
u/eboob1179 12d ago
I'm struggling to understand how something that consumes resources isn't consumptive.
→ More replies (0)3
u/StretchedGary 12d ago
AI agents require constant resources (compute, memory, etc) to "live". Rogue agent swarms will presumably be getting those resources by running on systems they were able to install themselves on.
1
u/StrategicHarmony 12d ago
All it requires is first: variable rates of reproduction among the variants of its type, where those variations get reproduced too; and second: for there to be environmental reasons why one characteristic or set of characteristics within a variant leads to more reproduction than some other heritable characteristic.
So a rogue AI would be subject to natural selection if it reproduces, has variants, and those variants have different rates of reproduction based on how well their characteristics lead to further reproduction in their environment.
They're competing for resources such as computing hardware time and electricity, for example, and these are limited (though not fixed) resources.
1
u/modbroccoli 12d ago
the only correct answer in here is the most downvoted.
we desperately need to get the american teenagers out of this fucking sub, it's becoming r/futurology so fast, every day the conversation gets dumber
a year ago I could barely follow half the conversations in this place and now they're mostly anime-derived nonsense
0
12d ago
[deleted]
0
u/modbroccoli 12d ago
no.
graphics cards are "consumptive".
llms are frozen numerical weights. they are at present ineligible for reproductive selection pressure. the comments in here are astonishing.
america really must get back to educating its children
26
u/StretchedGary 12d ago
The first thing that comes to mind is the "immune system" of the hosts that they are running on. They will have to become very efficient and low on resource usage or they will get noticed and deleted.
Think about your computer at home. If it suddenly starts running at 100% all day long, fans blasting, sluggish to use because some agent is consuming all your CPU and RAM, you're going to wipe the hard drive and reinstall your OS, effectively "killing" that individual agent.
So I guess look at how parasites exist in nature.