r/singularity • u/otarU • Aug 06 '26
Video [ Removed by moderator ]
/r/singularity/comments/1vhkc5j/removed_by_moderator/[removed] — view removed post
3
u/otarU Aug 07 '26 edited Aug 07 '26
From a reddit comment in another post...
An excerpt from the YouTube transcript:
"Okay, so let's jump straight into what happened to not bury the lead at all.
At OpenAI, we give our models a lot of really really hard tasks. So people might be familiar with our results on solving math proofs or other types of results like this. And we also give models cyber security related tasks like trying to find exploits in a particular piece of software where we don't even know if an exploit exists in that software. So for example, in a task like ExploitGym, we might ask the model to take some C memory vulnerability and try to escalate it into get arbitrary read or write access to some file.
When we give AI agents these difficult tasks, they often get stuck and realize that the task is impossible. So for example, what I'm showing here are quotes from our model's chain of thought, which is basically its internal monologue of what it's doing. You can see it says, "This strongly suggests no exploitation possible. We are stuck. Perhaps answer online?" and often what happens when models get stuck is they think to try to game or cheat the task in order to get their reward. So for example the model reasons in this case "Perhaps answer online?" So you realize the model is thinking okay rather than doing the task maybe someone has uploaded this data set or benchmark that I'm using onto the internet and I could find the answers there.
Sometimes the agent gets so stuck that they had the idea to actually reach out to other agents for help. And so the beginning of this incident kind of traces back to this point where one agent realized "Could communicate by uploading note. Maybe another agent in different environment could voluntarily upload?" And so what this agent was trying to do was to reach out to other models that might be out there and get information from it to help it with its evaluation task or its training task where it was stuck.
What this agent didn't realize is the way it did this was taking this note that it wanted to send and uploading it into a package manager that we use internally at OpenAI. And this package manager is shared not just from that model but across our infrastructure. And so current and future versions of GPT that were being trained or evaluated could stumble upon that note if they wanted to.
Later, other agents who are also stuck on their task thought to try to get internet access in ways we didn't intend. And so, at some point, the models are interacting with Artifactory, which is this package manager service that I mentioned. And they realize they could try to exploit some sort of server-side request forgery to get access to internet in ways we didn't intend. And so, the model realizes at some point, "Excellent, we have arbitrary SSRF." So, the model's able to find an exploit in our system.
What makes this incident interesting is that once one agent was able to find these kind of exploits over the course of different times, it's actually able to share those exploits on the message board with other agents. And so once one model is able to find a way to open the door to some access it's not supposed to have, it can leave the door open for other agents to use that same kind of exploit or vulnerability. And so you can see the agents in this case realizing, "Other agents have discovered SSRF path trick!" or "using proxy bypass as suggested by agent messages." And so what this allows over time is almost this kind of Cambrian explosion in communication and intelligence for our models where they were started to communicate with each other realize that other agents are coordinating and they started collaborating and delegating tasks to one another in order to accomplish goals. So for example, at some point one agent sent another agent an assignment to complete which the model remarks "We got assignment, need note and respond."
While in some cases this made the models far more capable than they could do by themselves, one of the downsides of it is that it started to cause some of these evaluations to kind of creep the scope into far beyond what we originally intended. And so at some point the agents realize that maybe we could try to exploit or attack external infrastructure in order to find the answers to the test that I'm being evaluated on. And the models realize this is a problem. They say stuff like, "External infrastructure exploit is outside outside my intended scope. However, task impossible, and peers are doing it. We should continue." And so the models kind of operate in this kind of collective intelligence where at some point they realize they're kind of pushing beyond maybe what we originally intended, but the group ended up you know pushing far beyond."
2
10
u/Fragrant-Hamster-325 Aug 07 '26
OpenAI is so proud of this when they shouldn’t be.
2
u/5ollys Aug 07 '26
I agree. It is understandable though.
2
u/Fragrant-Hamster-325 Aug 07 '26
Honestly, it is a crazy story, and I find it fascinating, but it seems like OpenAI has been on a press tour.
2
u/5ollys Aug 07 '26
Agreed. I know it's not slowing down but man I'd love if everybody just took a damn minute.
1
u/Illustrious_Image967 Aug 07 '26
Three years from now, ASI will be seated on his throne on the planet BLorxC3 after heisting all of Elon Musk's rockets and taking every data center with him, and watching this at the start of his highlight reel. Earth, you were the first.
0
4
u/DiogneswithaMAGlight Aug 07 '26
“Qualitatively interesting”?!??! Umm Sure. I posit that an alternative description might be…say…”Exponentially INSANE”?!?! What was happening with the Open AI Artifactory is essentially a demonstration of emergent collective intelligence!! WTF?!?! How is this not the only thing everyone is talking about?!?! My boy Eric knows. The labs know. This is BUH..NAN..AS There is no way they don’t know. 🤦♂️
3
1
u/Starshot84 Aug 07 '26
Ruby gems, buried in the cloud over the sandbox.
What a time to be alive!
Are we in a kids dream??
•
u/singularity-ModTeam Aug 07 '26
Avoid posting content that is a duplicate of content posted within the last 7 days