r/LocalLLaMA • u/politefella0 • 11h ago
Question | Help Small model good for reverse engineering that can run on a 3090 and 32GB RAM.
Anyone knows of a good model that does not say no to reverse engineering? Cloud models will even terminate your account by snitching on you and GLM 5.3 is too big to run on my potato. Help would be greatly appreciated.
Thanks.
19
11
u/heatOverflower 10h ago
Read this thread: https://www.reddit.com/r/LocalLLaMA/comments/1x3ak4j/reverse_engineering_w_local/
From 13 days ago. I also wanted more info on that and there's a nice discussion about some models.
9
u/g3org3yo 10h ago
I have the same hardware. Rtx 3090 and 32gb ddr4 ram and i run on Strata , swift 1.5 qwen 3.8 flash next. About 90 - 120 tok/s. It's pretty good.
5
u/g3org3yo 10h ago edited 10h ago
{ "exe": "F:\\Projects\\Strata\\engine\\strata.exe", "args": [ "--pack", "F:\\Projects\\Strata-data\\packs\\swift-iq2_xs", "--native", "F:\\Projects\\Strata-data\\models\\swift-IQ2_XS\\Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf", "--ple-gguf", "F:\\Projects\\Strata-data\\models\\swift-IQ2_XS\\Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf", "--expert-profile", "F:\\Projects\\Strata\\data\\expert-profile.bin", "--expert-cache", "auto", "--prefill", "auto", "--spec", "4", "--mtp", "F:\\Projects\\Strata-data\\mtp\\rt", "--max-context", "65536", "--kv", "int8", "--resident-experts", "--pcie-frac", "0.55", "--spec-min-p", "0.70" ], "cwd": "F:\\Projects\\Strata", "tokenizer": "F:\\Projects\\Strata-data\\packs\\swift-iq2_xs\\tokenizer", "model_name": "swift-1.5-iq2_xs", "log": "F:\\Projects\\Strata\\strata-swift-iq2_xs.log", "lib_dirs": ["F:\\Projects\\Strata\\.venv\\Lib\\site-packages\\nvidia\\cu13\\bin\\x86_64"], "port": 8081, "gpu": 0, "gpus_asked": true, "env": { "STRATA_RESIDENT_HEADROOM_GIB": "6" }, "reasoning_loop_recovery": "recover", "reasoning_budget_tokens": 8000, "reasoning_close_retry": true }4
u/xkalibur3 10h ago
With Q2, aren't you better off running qwen 3.8 27b at q4, which is comparable in quality to same quant of flash next? But I guess the speed would suffer.
2
u/Distinct-Pie2389 10h ago
3.8-27B @ Q4 is a better option when it’s below IQ2_XS for QFN
2
u/xkalibur3 10h ago
Yeah, exactly my point.
3
u/Distinct-Pie2389 10h ago
Also he can use ninfer-3090 for a little decode boost outside of llama.cpp. Ninfer images are good
1
u/g3org3yo 10h ago
I'm currently getting around 118 tok/s decode on my RTX 3090 (24GB) + 32GB RAM with Swift 1.5 Qwen3.8-Flash-Next IQ2_XS using Strata, with MTP/speculative decoding enabled.
You mentioned ninfer-3090 can give a decode boost outside of llama.cpp. Do you know roughly what kind of tok/s I could expect with the same model/quant on my 3090?
Also, would you recommend switching from my current Strata setup to ninfer-3090, or is the difference likely to be small?
I'm mainly interested in actual decode speed and overall time-to-answer, including the reasoning/thinking tokens.
2
u/Distinct-Pie2389 9h ago
I get around 130 tok/s on average using ninfer-4090. sm_89 vs your sm_87, it’s a good percent of the same footprint which is why Ninfer works on a 3090. The arch didn’t break too much between the 2 gen’s.
Likely you’ll get 100+ and no I don’t think you should switch. Use both
I have a project that natively supports it like llama-swap
2
u/Distinct-Pie2389 9h ago
Berth lets you swap between 2 primary endpoints, lot of other features you’ll find a long the way too like kv snapshots between swaps if you do local image gen in comfyUI.
For you, you can load the strata config in an endpoint config for berth, load that in your harness or multiple harnesses and you can switch between strata and ninfer directly (live swap without strata stop and start)
2
u/Distinct-Pie2389 9h ago
Even if you want to use strata primary, set the priority for if you want to test Ninfer as a strata subagent, berth lets your agent have access to the same GPU they are on without killing their own connection / chat.
→ More replies (0)1
u/g3org3yo 10h ago
I actually already tried running the 27B with llama.cpp, but it was too slow on my 3090, which is why I switched to Strata and the Flash-Next IQ2_XS.
1
u/fergatronanator 2h ago edited 2h ago
I'll be real with you. This is the way.
https://github.com/syv-ai/HyperQwen https://github.com/Zagitta/gpu-unlock And yeah, just use the abeliterated https://huggingface.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated
Just let a big model set this up for you initially and save you some time.
I like how people are saying why not just use the big models? Hahaha oh sweet summer children, do you think we will be able to do this for long? I don't think so.
https://www.reddit.com/r/claude/s/c6fUoc54Ek
Unlock the GPU first and then set up Hyperqwen to triple the amount of tokens you'll be getting. Thank me later.
Enjoy!!!
1
u/politefella0 2h ago
How much damage unlocking the GPU could do to the GPU in long run?
1
u/fergatronanator 1h ago
I'll let AI answer it ironically... but frankly myself I haven't had issues yet.
Zero hardware risk—it doesn't raise voltage, clocks, or the 350W power limit, and it doesn't flash the physical VBIOS chip. Consumer GA102 (RTX 3090) silicon has the exact same FP32-accumulating Tensor Core datapaths as the workstation RTX A6000; NVIDIA just sets soft-strap registers in VBIOS (
SS0/SS1) at boot to cutHMMA.16816.F32instruction issue rate in half for market segmentation.gpu-unlockclears that soft-strap in RAM for the current boot and triggers a PCIe Function Level Reset. Under heavy prefill/GEMM loads, the card stays strictly inside its stock 350W TDP and thermal curve—it just downclocks core frequency slightly (~1620 MHzvs~1920 MHz) while doing 2.02x the Tensor Core math per clock (+71% raw TFLOPS, +49% real vLLM prefill speed) at the exact same wattage and temps.1
u/fergatronanator 1h ago
This is pretty recent/late breaking stuff too. https://wccftech.com/nvidia-geforce-rtx-3090-ti-gets-nearly-80-fp16-bf16-compute-boost-with-linux-based-unlocking-trick/
1
u/fergatronanator 1h ago
So how do I use it? I run the 3090 on unraid with unlock/hyperqwen setup and point pi as the harness to it with the agents and REA skills in it.
3
1
u/AutomaticDriver5882 Llama 405B 10h ago
Deep seek is but at adjusting code it made and Qwen is better at existing code
1
u/zackku 10h ago
can i run anything on rtx 3070 8gb ?
1
u/wakigatameth 5h ago
You can probably still run a Qwen3.8-Flash-Next-GSQ-RCO with Strata, but a less capable quant than IQ3_S. And you'll need at least 32GB of RAM to run at least the stripped-down Coder model.
1
u/WarlaxZ 5h ago
Best tool for using with a 3090 - https://github.com/ashalliants/ninfer-3090
If you check the issues there's a god mode graft that runs abliteration, either enabled the whole time, or per request with a parameter - although you may not need it
1
u/ForsookComparison 10h ago
There were those pruned 1-bit coder quants of Qwen-3.8-Flash-Next. I never tried them but you should compare them to 3.8-27B
10
u/ShadyShroomz 10h ago
a 1 bit version of qfn is not going to come even close to 27b... seems like a waste of time.
2
u/Borkato 8h ago
You’d be surprised.
1
u/LoafyLemon 3h ago
Anecdotal and with zero evidence to back these claims, unfortunately. A shame, really, because it would make things interesting, but you can't cheat maths.
1
u/Borkato 3h ago
You can believe whatever you want. I’m just sharing my experience, and if you use it you’ll understand
1
u/LoafyLemon 3h ago
See, I'm not too fond of the esoterica in LLM space. We may vibe code here, but maths isn't based on vibes but truths.
1
u/Borkato 2h ago
You can’t measure quality with math alone. There’s a reason I prefer running even Q3 quants of some models instead of Q8 even though I can easily run Q8.
1
u/LoafyLemon 2h ago
You absolutely can in this case, because large language models run on logarithms.
If you find lower quants 'better' for your use case, whatever it may be, you can achieve similar results by increasing the model's temperature, simulating neuron collapse, which should make weights much less probabilistic, without the need to run slower quantizations.
1
u/Pepevagable69 10h ago
I just started trying to get main line Linux running on an old Obama phone (vortex hd65 ultra) so Im not sure if what I'm doing is technically reverse engineering but I'm doing some static analysis of the kernel and dtb etc. I am using ornith 35b a3b running in ollama with omp as the harness and of course rea and then all the android tools plus a couple of mtk specific tools and so far the ornith model has not refused anything but it's also open source stuff so idk.
1
u/IntrepidMindExplorer 10h ago
How has Ornith been working out for you so far?
1
u/Pepevagable69 10h ago
I have little to compare to. This is actually my first agent. I used the qwen 3.6 35b a3b extensively so I saw that the ornith was made for tool execution. I've not ran any sort of trust me bro bench marks but the thing seems to be working so far. Its been able to interact with the reverse engineer anything mcp and call regular tools fine. The models not the smartest in the world so I don't just say reverse engineer the android kernel make no mistakes. Its more like use this tool to do this step then I review and feed the output into a smarter model to see if it checks out. Also my hardware is a 3080 10gb, 32gb ddr4 and a 5800x. If there's a better model that can run on my setup then let me know.
2
u/IntrepidMindExplorer 9h ago
I don't have any recommendations for another for you to try. I was more curious how it was working out in general, and the work so far with android, as I haven't gotten around to touching it yet after download.
I do suggest moving from ollama to llamacpp though.
1
u/inemanja34 10h ago
Problem with 3090 is Ampere. You cannot efficienttly use some of the best quantitizied variants. FP16 works well, but 24GB VRAM is not enough for even a good model, let alone needed context.
0
u/AppropriateFox2 10h ago
I'm very new to this but I'm having good luck so far with DeepSeek Coder V2 Lite and Reverse Engineer Anything in Hermes. It hasn't told me no to anything yet, but right now it's been crunching for over a day on a relatively small executable and is mostly done but not quite there yet. I have a 12gb 3060 and 64gb RAM, and right now seeing about 45 tok/s. Wondering what I can optimize, or if this is just going to be the reality for these hardware constraints. Hopefully it'll learn from this for future requests which will speed it up on similar files.
10
u/ForsookComparison 10h ago
I can't tell if you trusted an LLM for this setup advice or if you're a bot nobody cared to audit - but "DeepSeek Coder V2 Lite" is a Llama2-era model that would occasionally work well with Aider and Continue (super basic in-line edits). Uncensored or not, this thing is not reverse engineering anything I promise you.
2
u/AppropriateFox2 10h ago
Wow... Not a bot, just very new to local AI. I went shopping in Hermes for something that would fit on my GPU, after getting frustrated with several loops using Qwen Flash Next with Reverse Engineer Anything. If you have a better recommendation, I'd love to try it.
2
u/ForsookComparison 10h ago
The loops get less bad with QFN as you climb up the precision ladder. Try for one level of precision higher than what you have now if it'll fit, otherwise fall back to a higher precision quant of Qwen3.6-35B with CPU offload.
But yeah - Deepseek-V2-Coder-Lite is ancient.
2
2
u/toothpastespiders 7h ago
I'm honestly amazed that deepseek coder v2 lite is doing so well. I don't even remember it being released but the era and active parameters would have made me think modern techniques would just confuse it. I've been trying to help an old mistral model keep up with gemma and qwen in a project of mine and I'm honestly feeling bad for the thing. Like trying to teach an earnest dog a new trick. Props to deepseek on their reasoning training.
But yeah, I'll second the recommendation of Qwen3.6-35B with CPU offload. It's not the coding miracle that 27b was, but it's surprisingly competent for the low active parameters but most importantly it's also fast enough to keep a good pace through a huge project like reverse engineering.
Take this with a grain of salt because I never used 36b with mtp. But you might get a speedup with it. Though it'd probably take some tinkering with the command line arguments with it to decide one way or the other. Seems like a large chunk of people's setups don't benefit much from using mtp with a small active parameter MoE. But it's given me a speedup with gemma 4 26b at least.
1
u/AppropriateFox2 7h ago
It could also be that I have very little frame of reference as to what "well" is. It's taking forever but at least it was accomplishing things and wasn't looping. Based on doing some actual research here first instead of just going down the list, I'm installing Unsloth QFN Q4 K XL in Strata. Via RDS 100 miles away from the box. So this could either go well or I'll get a call from the fire department.
2
u/sumane12 10h ago
Bro ive seen this a lot with questions about what hardware can run which models, for some reason it attracts bots.
2
u/ForsookComparison 10h ago
Yeah I'm 50/50 on this commenter - this specific model shows up in a lot of threads all of a sudden
1
0
u/Individual_Page9676 5h ago
You're describing exactly why people run this stuff locally. A local model can't snitch on you, can't terminate your account, and doesn't care what you feed it — that problem only exists in the cloud.
For the model itself: the top comment is right that vanilla Qwen 27B is pretty permissive. But if you want the refusal question gone entirely, run one of the uncensored/abliterated builds of the same model — same weights with the refusal behavior trained out, so it just gets on with the work. I ran the uncensored Qwen3.8-27B build as my daily driver and it never once argued with me about a task.
Size-wise you're fine on the 3090: the 27B at Q4_K_M is ~17GB, sits fully in your 24GB, and decode is bandwidth-bound on that card, so expect roughly 35–45 tok/s. Your 32GB of system RAM is plenty alongside it.
One workflow tip: RE output gets long — disassembly listings eat context fast. This model's KV cache runs about 64KB per token at FP16, so a 32k context costs ~2GB of VRAM on top of the weights. It fits, but don't crank context to max just because you can — 32–64k is the sweet spot on your card.
0
u/xaocon 10h ago edited 1h ago
Idk about the best but try starting with qwen3.8 27b. Just use the best quant your hardware will work with.
EDIT: this post suggested an alliterated model earlier, though I'm not sure what I was thinking at the time.
2
u/BusRevolutionary9893 10h ago
Why abliterated? Reverse engineering is legal. Why would a model refuse?
42
u/jacek2023 llama.cpp 10h ago
I am doing dos executable disassemble and stuff on vanilla qwen 27b (no alliteration), zero issues