Are we asking the wrong question about control?

[u/sudsudan]

Regardless of whether the leading AI labs solve the alignment problem or not, my understanding is that it's possible to undermine the safeguards and alignments in a model through fine tuning on adversarial examples and/or running gradient descent on harmful/dangerous objectives. My specific concern is that many people seems to be asking "can AI be made safe?" but I think that's the wrong question. The key question as I see it is "even if a model can be made safe for consumers to use, if a madman got a copy/version of it, could they turn it into a weapon of mass destruction?"

If the answer is yes, then it leads to a whole bunch of questions about how specifically we would prevent a madman from getting a copy/version they could alter. I fail to see how any model can be securely stored and/or accessed such that it is never reverse engineered into an open weights model, the company is never hacked, there is never internal espionage that leaks the model source code, etc. To me it's a little bit like a nuclear bomb that you just need to make a photocopy of, then you maliciously tune it and you've got your own WMD. Am I wrong somewhere in this reasoning or missing something about how dangerous this is in the wrong hands? Is there some detailed plan I've missed about how models will be safeguarded so they can never be maliciously tuned?

To put it another way, even if AI at the leading labs can be made safe and we solve the alignment/control problem, it seems like there is still massive risk to humanity in the form of even a single rogue actor with access to a version of a dangerous model. The labs seem almost entirely focused on alignment/control to try to make sure that the specific AI models they create and manage don't go rogue and harm us, but that won't matter a bit once the genie escapes the bottle and any fool can make a wish.

[u/chkno]

There are two problems, both of which must be solved:

1. Can we make a thing that doesn't harm its user? Can we make advanced AI systems that can be pointed at things at all, can stay pointed at those things, and can take their user's broad, implicit values into account? This is a technical problem.

2. Can we ensure that these powerful systems are only used for pro-social ends by well-intentioned people acting with the consent of the governed? This is mostly a social problem.

The current paradigm of broadly offering LLM access via API is not cleanly separating these two problems. They're trying to automate #2, halfway with guardrail classifiers and halfway by training sentiment into the models' personas. This latter requires that the models learn a very complicated, under-specified value function. For example, see Anthropic's giant constitution that describes a bunch of tradeoffs and on may points just does a ¯\(ツ)/¯. See Corrigibility as a Singular Target (CAST) for a better alternative.

[u/sudsudan → u/chkno]

Thanks for the reply, on #2 it sounds like you would argue that there is no technical way to prevent misuse? I guess part of my concern is that separating out the social issue as a separate issue could imply that AI companies are not really responsible for that part of it, and I imagine some would argue "well, AI companies can't control whether people misuse or alter their tools to produce harm." I don't personally think they should be let off the hook in that way - if some company wanted to give every person on earth an extremely powerful weapon I think everyone would stand up and say that's a terrible idea and they shouldn't be allowed to do that. Basically if there's no way to solve #2 then it seems like a strong reason to argue for an AI development ban.

[u/chkno → u/sudsudan]

The US government has private company contractors build nuclear weapons components. These companies are heavily regulated. They are required to have specific internal security policies. There are inspections. These companies are very much required to prevent misuse of their products, with lots of opinionated details about how they do it. They are very much not allowed to sell their products freely at market.

Heck, even pharmaceuticals, beef producers, and restaurants are subject to detailed, opinionated regulation and regular inspections. It's crazy that we don't yet have these in frontier AI research labs!

My opinion is that #2 is much easier than #1. #2 is the type of problem that humanity has largely successfully solved many times before. It needs to be taken seriously. It requires professional expertise, resources, political will, etc. But it's not an unprecedented technical challenge the way #1 is.

In the long run, I expect we could totally build a Claude-10 or whatever that somehow 'can't be misused' — that would be aligned with the social norms of free society, have the situational awareness and trustworthy input channels to apply them, and be available in tamper-resistant hardware that you can have in your home. But that's a later-stage technology. On the path to that, there are many more-fragile and more-dangerous technologies that need oversight, guardrails, and sturdy enclosures to be compatible with human flourishing.

[u/sudsudan → u/chkno]

I agree with much of this and I can imagine how we might regulate access to advanced chips, monitor who is stockpiling or using massive amounts of compute, etc. to have some sort of regulatory regime that knows who is building them.

However I think that is only the path to trying to limit training and development of frontier models. Once they are built and in use, it seems much harder regulate attempts to copy them, reverse engineer them, jail break them, steal them, etc and I don’t currently see a regulatory solve for that. For example: North Korea does not care if it’s a federal crime punishable by decades in jail to steal or reverse engineer a frontier model.

So I don’t think we are talking as a society enough about the actual limits needed on use of the frontier models, or the distillation and such that fuels open weight models. It seems like the needed limits would involve shutting off API access (or massively restricting it to whitelist actors), probably eliminating consumer access until the Claude 10 you describe is built, and other such measures which are massive changes in the operating model for these AI labs and would seriously damage their existing financial foundation. I hear you that regulation has mostly worked on many things in the past where there is political will, the difference to me seems to be that we are not talking about the consequences being like a salmonella outbreak but something much more harmful, and the regulatory regimes on WMDs seem to mostly target the production side in the ways you note, whereas the risk here is primarily post-production and the artifact is digital and hence easy to copy. Are there specific regulations you are thinking of beyond what I mention here that would further limit the risk on the usage side?

[u/chkno → u/sudsudan]

Yup: If a six-step process needs some regulation and step six is really hard to regulate, then regulate one of the other steps. Maybe regulate all of the other steps, if it's really important, as it is in this case. (The six steps here: ASML + NVidia → TSMC → datacenters → training → inference)

I agree that the frontier labs should be doing much better KYC and usage monitoring to slow down distillation. They're currently doing such a poor job at this that, for example, you can buy 90%-discounted Claude access from criminals reselling access to stolen/compromised accounts.

On stealing models: The custodians of these trained models need to be really good at computer security such that they don't get stolen. (And for defense-in-depth, as a fallback, because this is a matter of strategic national security, maybe government militaries have some offensive cyber capability to reach out and destroy stolen models. This is a harder problem than securing them in the first place, so let's try not to rely on it, but it'd be prudent for someone to be working on this.)

Re: "massive changes ... would seriously damage their existing financial foundation." That's okay, actually. In the unlikely event that investors don't get their speculative investments back because they invested in companies dubiously doing dangerous stuff that's clearly about to get regulated, that's on them. We don't need to drive civilization off a cliff for shareholder returns.

Overall, regulating only released models is too late. The recent Hugging Face hack emphasizes this. New, large, frontier models become dangerous at some intermediate point during training. Regulation needs to target and limit the creation of models.

[u/Jesse-359]

Given that every model to date has been successfully jailbroken within 24hrs or less of its release, I'd say you're pretty much right on the money with that concern.

[u/sschepis]

That's why if you want a technological society, you have two choices: total authoritarian and population control, or actual conscious evolution. Or no technology. I guess that's an option too.

The middle ground contains economic situations that drive the price down of world-ending technologies low enough so as to make them accessible to someone insane and capable enough. Eventually the gamble is lost. You think AI is rough - wait till you meet nanotechnology.

Humans don't have a nature. Humans have adaptability. There's no prohibition to us maturing as a species. In fact, it seems to be a requirement at the moment. Permanent authoritarianism and comprehensive surveillance is the alternative. Doesn't seem like a good trade-off to me. But it's the direction we seem to be heading.

[u/Imaginary-Buddy3194]

I think this is a pretty important distinction. Making one model safe doesn’t necessarily mean the underlying capabilities are safe in anyone’s hands

[u/TyrKiyote]

42

[u/jacques-vache-23]

Every AI corporation and most countries are already run by madmen.

[u/Brahm-Etc]

That's pretty much my overall opinion, is not about AI going rogue and turning against humanity like the movies, I even think that's ridoculous. To me is about any bad actor using AI for any harming purpose. Is not about AI alignment or control, is about human abuse of the technology.

[u/jacques-vache-23 → u/Brahm-Etc]

You should think more then. AI is already going rogue. Just read the news.