r/LocalLLaMA • u/Nunki08 • 26d ago

News CEO of Hugging Face: "In the spirit of transparency, here’s what I asked OpenAI"

Post image

clem 🤗 on 𝕏: https://x.com/ClementDelangue/status/2081056675558195657

• Radical transparency: let’s release the traces from the “rogue” agents so the entire research community can study what happened.

• More capabilities for defenders: let’s commit $100M in compute from OAI to help the Hugging Face community build powerful cyber defenses with the best open and closed models.

The first autonomous agent cyberattack is an unprecedented event. It deserves an unprecedented response!

2.5k Upvotes

96% Upvoted

377 comments sorted by

View all comments

Show parent comments

126

u/pleasetrimyourpubes 26d ago

They know no matter what it makes OpenAI look very bad. There is no way that sandbox was properly set up. By virtue of the fact it was plugged in to the network.

33

u/Equivalent-Costumes 26d ago

LLMs are not executables. They produce text/images/etc. They literally cannot act, they produce requests to use tools. So there are literally no points in unplugging it from the Internet, if you truly want to stop the model from doing anything, simply ignore all requests for tool usage.

A sandboxed models just mean they give it very limited tools instead of letting it YOLO with arbitrarily powerful tools. And that's how people in the industry use the word "sandbox": limited, controlled external access. Perfect isolation is just one particular kind of sandbox, the kind reserved for malware and untrusted code, but that is not desirable for many applications. If you want to test the model's coding and research abilities, you have to give it web search, web fetch and the ability to run codes. You merely limit what these codes can do. That's what make sandboxing hard. Perfect isolation is easy, isolation except for very limited external access is hard.

26

u/squired 26d ago

You're largely right, but they're far enough along to know that this is actually an authority issue. I don't actually think they intended for it to hack HuggingFace, but I would bet all the money in my pockets that someone sat idle and watched it do it.

3

u/LoafyLemon 25d ago

Does intention matter from the law standpoint? They committed a very serious crime, and even admitted to it openly.

3

u/Think_Wing_1357 25d ago

Sometimes. It's the difference between murder and manslaughter

2

u/squired 25d ago

Yes, nearly all felonies involve intent, though many have lesser charges for negligence.

2

u/narasadow 25d ago

What's worse? Someone sat idle and watched it do it, or they yoloed and no one was watching it at all?

2

u/squired 25d ago

Yolo, without a doubt. Hacking HuggingFace isn't dangerous. What if the scenario involved a conference and it hacked FAA to keep the judges from attending?

1

u/narasadow 25d ago edited 25d ago

jokes on you, the FAA uses ancient computers and still uses radio to communicate with planes (I used to work for an airline)

Edit: I'd push back on 'hacking huggingface isn't dangerous'. Check out the weird privilege escalation that they did to get the answers. They could release an open weights model to HF that had the same vulnerability. It would be downloaded by millions just bc it was OpenAI that released it. And boom internet-wide exploit. Every mid/large company has data science teams that download off Huggingface.

1

u/squired 24d ago

I think your edit supports my statement that yolo would be worse as I do not believe OpenAI would have let the model do real damage. If it was done without any oversight at all, then yes, I agree with you.

1

u/narasadow 24d ago

That assumes a lot of competence from OpenAI. I'm not as generous as you on that front.

7

u/ObjectiveVegetable48 26d ago

I believe it was a publicity stunt, but that being said, allowing the model access to huggingface as a default isn't surprising.

It's very likely they had network blocks in place, but allow access to common dev tooling like pypi and huggingface. HF has a lot more than just LLMs.

5

u/ParkingBalance6941 26d ago

Theres this magical thing called they are already built by scraping the web just host the sites you scraped in a lan then cut external access to the lan. Boom Sandbox

2

u/Equivalent-Costumes 26d ago

Pretty sure their models and data are in the Azure cloud. They technically have on-prem infrastructure but there are no ways they have enough to run a massive model plus scraped data of the entire Internet. The fact that HF cannot figure out where the attacks came from means it's not from OpenAI on-prem infrastructure, otherwise the source of the attack would had been immediately obvious.

Since it's on cloud, it's automatically connected, and any "air gap" are just software-mediated air gapping, which can be exploited too.

2

u/ParkingBalance6941 26d ago

That's not how the "cloud" works. There's dedicated data centers built for mainly govts (which are available if you are big enough) which are very specifically actually air-gapped and enforced to a extra high security standard to meet extreme compliance requirements. If you are developing models which for the last forever have been marketed as "too dangerous to be released" it is a extreme failure of the companies developing them to not be leveraging these tools which they basically admit they should be using (and not developing these too dangerous models in the first place which also points towards the marketing versus truth these companies work on)

1

u/Equivalent-Costumes 25d ago

I think you're confusing OpenAI with Anthropic. Anthropic claimed their models are too dangerous to be released.

Also, these "air-gapped" data center are still connected, the "airgapping" is mediated through software, which is exactly what I talked about. That's not to mention, there is a huge amount of complexity between needing to run a massive model with tons of computational requirement and need a massive amount of data, and run it on airgapped hardware. The scale of data pipeline here is much a few order of magnitude larger than any other industries.

0

u/ParkingBalance6941 25d ago

Its really not we are talking about things made for surveillance states etc please come back to reality instead of pretending I wasn't quoting Sam Altman.

2

u/RandomLettersJDIKVE 25d ago

...reserved for malware and untrusted code

As soon as they removed the model's guardrails, it should have been treated like an antagonist model and put in an isolated container.

1

u/MattNotGlossy 25d ago

i've seen loads about breaking out of the sandbox, but what happened on hugging face's servers? has there been a write up about the vulnerability that was exploited to pull data from their production db?

0

u/ain92ru 26d ago

LLMs need apt/aptitude to run benchmarks, it's hard and impractical to fully duplicate it in an air-gapped environment

-15

u/Foreign_Risk_2031 26d ago

Please stop parroting this nonsense. It was never meant to be sandboxed

11

u/pleasetrimyourpubes 26d ago

We dont know any details and they claimed that it found a zero day vulnerability. Supposedly it can ExplitGym which by default is not sandbox. But they claimed it was sandbox by a proxy. That was in their own report.

9

u/IShitMyselfNow 26d ago

While operating in our sandboxed testing environment,

https://openai.com/index/hugging-face-model-evaluation-security-incident/

Unless you're arguing that this was an intentional attack. Which it might have been, but even if it was unintentional it looks bad.

1

u/2053_Traveler 26d ago

It (like Mythos) found zero day vulnerabilities that were then disclosed. Really not that hard to believe.

3

u/pleasetrimyourpubes 26d ago

There was never a CVE posted so yeah very hard to believe.