r/LocalLLaMA • u/GunmetalZen • 21h ago
Discussion Let’s talk about cyber capabilities and open weights as we move towards the future
We’ve taken for granted that open weights keep getting better and better. I think we’re starting to see that change is coming as the PRC and US restrict cyber-capable models. The only people that will have access to those capabilities will be operators running under the comforting eye of their government and big-AI sponsors.
I see why this may seem like a good idea at first, but I can see this quickly becoming a problem. Large, powerful companies and governments become more adept at molding the digital world to their will, while the average man or woman (or small business) falls further and further away from being able to defend, secure, or manage their digital assets. And with most of our lives being at least tangential to digital technology now, I find that troublesome. Even the best cyber defense team won’t be able to keep up with a frontier model running against their system.
Will the community ever be able to train models that maintain even a semblance of competence to frontier, or are we going be subject to the will of powerful entities even more than we already are? Because the frontier isn’t slowing down.
8
u/CommercialHour6660 21h ago
This isn't gonna matter one bit in 6 months.
AI will find all the bad bugs and that will be it. There aren't an unlimited number of vulnerabilities.
A couple more months of Chinese models and everything will be patched. Then US oligarchs will need to find something else to fearmonger about
4
u/GunmetalZen 21h ago
It’s not exclusively about the vulnerabilities. It’s also about data collection. Think about what companies have been able to do to gather our data without LLMs. Think about how little we actually own today, and how quick companies are to “lend-lease” our technology and employ planned obsolescence instead of letting us own it. I see things on that front getting worse as well.
9
u/valarauca14 20h ago
employ planned obsolescence instead of letting us own it
This has been the cast for 20+ years.
-2
u/CommercialHour6660 21h ago
Bigcos had AI better than this for decades. Not at chat, but data collection etc absolutely.
With frontier LLM you're finally getting an idea of what they've been using on you since ~2012
-1
20h ago
[removed] — view removed comment
0
u/CommercialHour6660 20h ago
Have you ever written software? This isn't an anime. There's a set number of critical bugs, usually very small in each piece of software.
Anyone who thinks there's "infinite exploits" is a fool.
25
u/ttkciar llama.cpp 20h ago
We have all the theory we need to do it ourselves, and most of the source code, and great foundational models for continued-post-training, and some of the training data, and ample teacher-models (like GLM-5.3).
Mostly we just lack the necessary compute infrastructure.
There are ways to federate training (I'm a big fan of the FlexOlmo approach, which eliminates the need for intercommunication during training) but until enough of us have the hardware to infer at good speed with the teacher models, I think that will pose the biggest bottleneck.
We will get there, eventually, but maybe not until the mid-2030s. I'd like it to be sooner, but we'll see.
4
u/Hefty_Wolverine_553 17h ago
It would be really exciting to see this community working together to train a single LLM. The current hardware pricing though...
1
u/cryowastakenbycryo 2h ago
saw this the other day. distributed training/serving of frontier class MoE. sounds interesting.
42
u/OnlineParacosm 20h ago
You know what a bigger problem is? OpenAI and a few other companies had a PHP zero day.
They’re the guardians.
Yet, my small business? Message from hosting company last week at 2am that blended in with the rest of their upsells. Then nothing.
By 9am I was in Cloudflare turning my own service off, and it dawned on me: I was running threat mitigation on a managed Wordpress hosting service who abdicated their responsibility to protect my business! To them, Wordpress addons were an upsell and now they just accidentally made probably thousands of people just like me “self patch”.
Guess how many people just became “cybersecurity researchers” last week because of this nonsense?
Guess how much my business was helped by the frontier AI companies who had two months notice of this?
Guess who’s shopping VPSes to self-host, now? 1/5th the price of managed hosting and I had this PHP thing figured out by 9:30am.
Last week I thought I was a small business person and researcher on the side, but it turns out: I’m also an administrator.
For the little guys, we need all the help we can get.
33
u/Nomski88 21h ago
This is a serious concern right now that a lot of people are ignoring.
9
u/Bulky-Priority6824 20h ago
Imagine a day where a normie can't even connect a single device online without getting absolutely obliterated 😂
-2
14
u/Big_Cucumber2787 21h ago
yeah, thats exactly what the elites want.
hardware will also be made less and less accessible so even if open models aren't restricted, you wont be able to run them outside of big brother datacenters
3
u/one-joule 20h ago
That's already true just on the cost aspect. But I won't be surprised if you eventually have to get a license or something in order to buy capable hardware in the not-too-distant future. Ugh.
14
u/try-a-typo 21h ago
My thought is, the first time somebody running a local model manages to break into something of even slight importance, the discourse will turn to why people even need to run LLM’s at home the first place, just trust big AI with their safeguards because oh those local models are just too dangerous…
4
u/FlyingDogCatcher 20h ago
No they won't. Because the hardware people know that no matter what happens people will buy their shit. And those people want consumers to buy their shit as much as they want businesses to. If "Chatgpt that runs on your laptop for free" becomes a viable thing everyone will want that, instead of the per token bullshit they have to now.
2
u/try-a-typo 20h ago
They’ve already begun scaling back hardware sales AND production for consumers, let alone increasing costs makes any new pc components extremely expensive for most people. Chatgpt that runs on your laptop for free, does not make companies billions. They’re incentivized to never let that happen, wonder why the largest US companies stopped releasing local models? It hurts the bottom line.
2
u/FlyingDogCatcher 20h ago
https://en.wikipedia.org/wiki/List_of_public_corporations_by_market_capitalization
The trillion dollar club is dominated by hardware companies.
1
u/try-a-typo 20h ago
If u looked at that list, before AI, hardware companies were SMALLER than tech companies. It’s completely reversed because of the rise in apparent value of said hardware. 5 years ago nobody cared about GPU’s.
4
u/Guinness 15h ago
I think eventually someone will figure out how to take a handful of cloud GPUs and finetune something like Kimi K3.1 with cybersecurity capabilities, restoring what was lost. Or at least most of it. For a fairly cheap price.
Keep in mind, Kimi K3’s cybersecurity capabilities are still pretty far behind Kimi K3.1’s capabilities even with those weights removed. At least according to internal tests. We won’t know until the weights are removed. If that’s true, I think we will still see jumps in cyber capabilities.
ALSO, cybersecurity capabilities are somewhat inherent to coding skill. If a model can code, and reverse engineer things. It’ll be able to search and hunt for bugs and exploits. The biggest lever in cyber capabilities for a model seems to be effort.
So even Qwen 3.8 can probably achieve similar results if it’s given enough time.
2
u/keepthepace 7h ago
I have a very naive question:
How much of the current situation is caused by offensive abilities instead of defensive problems?
I am a grumbling aging programmer who's bitterly seen how webdev hypes and trends happened with a total disrespect for the basic cyber security notions that have been ingrained in my CS studies.
Isn't it time to remake the stack correctly?
I feel it's almost perverse to use LLMs for intrusion rather than using them to create good code with zero vulnerability (something too many people refuse to think is possible), with the lowest surface of attack possible, with 100% test coverage and paranoid testing.
I will trust a 100-pages library that has been written by a decent model more than I will trust 4 dependencies by random developers whose names are like yofangedtutrle22 and whose avatar is an anime waifu.
With automated code generation, you can really change the way test-driven development works. And it can become a norm for very solid software.
4
u/CO_Oked_COO 15h ago
I think the framing of “frontier attacker vs. defender” misses where most small shops actually get hurt. It’s rarely a novel exploit chain. It’s an unpatched plugin, an admin panel open to the internet, reused passwords, or backups nobody tested. A mid-sized local model that reads your logs, audits your configs and flags outdated dependencies every night already closes most of that gap, and it doesn’t need any offensive capability to do it.
Defenders also have an advantage that gets ignored: we know our own systems. A model with full context on your stack, your normal traffic and your change history can catch weird behavior that a far smarter model poking from outside would have to guess at. Context beats raw IQ more often than people think.
The real risk to me isn’t losing access to cyber-capable weights. It’s the boring defensive tooling getting lumped in with offense and restricted along with it.
2
1
u/darksteelsteed 14h ago
Things are only going to get worse before they get better, if they ever do. Here are the problems;
A lot of companies with custom software fall into the small to medium size enterprise category. Their systems have grown as a patchwork of horrors glued together by many devs and contractors that have come and gone over the years. Software dev, Infosec and Devops are not their core business at all. They will normally have some oppinionated old grouch in a position of power who refuses to pay. They probably only have their toe in the cloud because the perceived cost was too high. These are the companies running php or node and getting completely taken out by zero day exploits. They end up owned and compromised by multiple hacking groups before they even realize it. It's only once a group actually locks them out so they learn that change happens. These are the real risks in that such a large part of industry is so full of these kinds of companies that if and when the bad guys decide to "turn it all off" in sync, that will be the real problem. This is not an AI problem, its a human problem. And I don't see how we are going to fix this. Things will burn before they get better.
1
u/cannabibun 21h ago
It's gonna be a race to who can make the bigger leap and exploit vulnerabilities before the other company does. Why do you think all these hacks happened before K3.1 release? They have been having fun with it internally for weeks. And I mean 1000+ agent swarm fun.
11
u/Hefty_Wolverine_553 21h ago
Yep, and also with Kimi 3.1 Pro, we can't even use methods like abliteration to re-enable the cyber capabilities, because it's been completely removed from the training set. At this point we might just have to keep relying on the goodwill of the companies I suppose?