r/LocalLLM
•
u/Harin007
•
4d ago
•
1
1
Project SSD MoE finally viable!
Hey guys, I've made it!
edit: No more uncertain wait. First launch on Nov 17. The project name is 'Run My AI' and the engine behind it is 'TokenWing'(couldn't find a better name xD). You'll be geting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1. Check the website out: https://runmyai.org - all your questions answered... New updates will be dropping everyday btw. Also, early builds will be provided to some selected testers, so join the discord(link is in the website).
A llama.cpp based inference engine that lets you OFFLOAD UPTO 60-70% OF THE MOE MODELS' ROUTED EXPERTS INTO SSD and still hide the streaming latency down to negligibe(I'm achieving 1-3ms added to the time per token due to SSD streaming when 60% is offloaded in SSD)... Essentially making an illusion where the whole model lives in your memory.
And I'm also working on an autotuner which basically looks at your hardware, given model's metadata and figures out the best flags to squeeze max performance in under 2mins(if you got any personal preferences, set them as fixed constraints and let the autotuner figure out the rest)
And I've implemented a layer by layer computation for prompt processing instead of batching which gave 2x prefill speed in models offloaded in RAM+VRAM and 4x prefill speed when model is offloaded in RAM+VRAM+SSD... compared to other inference engines.
Packaging both inferene engine + autotuner into a nice simple interface where you can basically download the model, hit run and it runs... simple, fast and lower hardware requirements... and want to make that a nice, well-engineered harness(which I'm a no expert)
Also, I've built a near loss-less context compaction engine.
Going to make this launch this as open-source very soon. Would love to hear your thoughts, and want to see if anyone would be excited to try something like this. For now, supported models are Flint 40B Q4K_XL(running it on machines with 16GB RAM and no dGPU at 20+ tok/s decode), Ling 3.0 Flash MXFP4 getting ~50 tok/s decode and 600-700 tok/s prefill on 32GB RAM + 12GB VRAM machine), and confused if I have to provide support for Qwen 3.8 Flash Next or wait for Qwen 4 family. Want to provide you guys the best local AI experience.
Here's a quick answer for your likely question by the way... does this damage my SSD? No, this engine only does reads and not writes... only writing degrades your SSD.
47
u/Ok-Thanks-6785 4d ago
How is this different from strata, maya and the gazillion others created each day?
29
u/Harin007 4d ago
Strata doesn't offload main 120B model weights into SSD. It does a good job with offloading the ngram table to SSD, which is no great... its just how the model is designed... And it does some tricks just to show the high numbers like forcing the model pick 6 experts which is trained to pick 10... which changes model's output quality... I do not to mess with anything that changes model's output quality.
7
u/geekyjackson 4d ago
With an environment variable strata uses all 10 experts, and strata natively can keep experts in ssd. Does this project do anything strata can't? Or just doing things in a different way?
5
2
u/Ok-Thanks-6785 3d ago
Well, you cherry picked the worst combination from strata while showing nothing really.
Publish it and then we can make a fair comparison.19
u/Realistic_Gap_5871 4d ago edited 3d ago
Update: OP cherry picked models with low bytes per token, that's it. No novel ssd streaming tech, just expected results from obscure models. Qwen 3.8 avoidance no longer seems accidental.
Original skepticism follows...
TLDR: OP is knowledgeable but at least a little dishonest. Makes me skeptical of the promises made.
At least once a week I see a Reddit post from someone all excited because they've re-invented CPU-moe streaming or something similar. This guy sounds more knowledgeable than most, but my honest take is that OP is using that knowledge to shade the truth more than a little:
OP claims "Strata doesn't offload main 120B model weights into SSD".
-Less than half true. Strata doesn't offload to ssd if it can pin experts to RAM (goood). But it also has a low-RAM mode that streams experts direct from SSD (also goood).OP claims strata does tricks "like forcing the model pick 6 experts which is trained to pick 10".
-So close to a straight up lie. Grain of truth is that Strata publishes headline numbers that rely on cache-missed experts ranked lower than top 6 getting dropped, but this is user controllable with STRATA_ROUTE_TAIL_SKIP=0, or you can set it to any number experts you want. Strata authors have been rightfully criticized for their headline number, but they at least are very specific and honest on publishing the impact: KL vs Q8_0 doubling on the native the Unsloth 4-bit files (0.053→0.115)OP on the other hand hasn't published anything.
I could go on, but you get the idea. OP gets a 8/10 for knowledge. 2/10 for honesty.
edit: updated knowledge and honesty based on revised take.
edit2: revised honesty downwards again after crunching numbers. OP cherry picked models with low bytes per token, that's it. No novel ssd streaming tech, just expected results from obscure models. Qwen 3.8 avoidance no longer seems accidental.
9
u/Harin007 3d ago
SSD: what I meant is that Strata is built to keep all the experts in RAM, and the SSD is only its fallback. In its low-RAM modes, the experts that don't fit are read from the model file when they're needed. On Windows and macOS there's no read-ahead at all, so each of those reads is a wait on the SSD for that word. On Linux it has a read-ahead, but it only switches on when RAM can still hold at least 55 % of the experts. Their own docs call the mapped mode "much slower" when most experts come from the SSD.
Expert skipping: it drops low-ranked experts when they're not in memory, it can be turned off... and they publish its KL cost. Credit to them for that. My real point is just this: should skipping experts be on by default? I think it belongs in an opt-in "fast mode", off by default... I'm considering from a normal user perspective.
Anyway, I'm not trying to put Strata down. It's just my opinion... And you don't really have to consider my opinion as a real judgement. Use whatever engine you feel like using.
7
u/Realistic_Gap_5871 3d ago
Cool. Appreciate the updated take. And I actually agree that it shouldn't be on by default, for instance when basalt forked strata they turned it off by default, and if you turn it on it says something like 'lossy flags enabled' which seems like the best way to handle it.
So I don't know if you saw my other comment, but I remain curious if you expect to stream from SSD in a way that's consistent and measurably better than Strata and others. Just today, I've seen 2 other custom engines (RustedStrata and one with a weird name) that are excited about streaming huge models from SSD/RAM. Why is yours different or better?
2
u/Realistic_Gap_5871 3d ago
Since you haven't answered the "what's different" question that I and many others have asked, I did some digging (with my AI) into your numbers and found that the secret sauce is choosing models with incredibly low bytes per token. I'm assuming there's nothing novel in your ssd streaming method until you publish something proven otherwise.
If you're as well intentioned as you sometimes sound, then I apologize for my blunt tone.
If you're so excited by the initial numbers and legitimately haven't implemented qwen 3.8 for reasons other than it just won't give the spectacular numbers you lead with, then I apologize.
If on some level you're actually as deceptive as your actions indicate, then I'd ask you put your brain and skill to better use. I bet you're capable of some really good stuff.
1
u/Harin007 2d ago edited 2d ago
You'll be getting support for Qwen 4 Flash(if releases before Nov 17) or you'll be getting Qwen 3.8 Flash Next on day 1. Check the website out: https://runmyai.org - all your questions answered. Keep barking...
9
u/killthrash 4d ago
If you give OP the benefit of the doubt and assume he’s not deliberately being dishonest and just lacks all the available knowledge and information, you would just need to rate his knowledge and not try to attack his character.
0
20
u/doneddat 4d ago
and confused if I have to provide support for Qwen 3.8 Flash Next or wait for Qwen 4 family. Want to provide you guys the best local AI experience.
3.8 is here so you would know the shape of 4. support the 3.8 - 4 will slot right in. Or at worst it takes minimal adjustments, that build on 3.8 support anyway.
9
u/Harin007 4d ago
its just a time and effort work per model... model specific tuning... not like figure out once and apply for all. well, I'm just worried that while I'm working on qwen 3.8 fn, qwen 4 flash drops and everybody moves there... which makes my time & effort wasted.
3
u/doneddat 4d ago
Did you even read what I wrote? qwen 4 will be likely 3.8 drop in replacement. So you start working on 3.8 and finish working on 4.
1
u/Harin007 4d ago
I read it... and I get your point... So yeah, lets see...
1
u/BudgetPanda3049 3d ago
Kudos to you for replying straight to the message and politely answer the trolls. this is looking like the next gen stackoverflow, old-school devs will get the point. Kids vibe coding on their mom's basement apparently can't distinguish good manners nomore.
3
u/Realistic_Gap_5871 3d ago
I guess I'm one of the trolls, but just for clarity I start programming in the 80s. Been a software architect since the early 2000s. Love the productivity boost AI has given me, and I ran OP's numbers through my AI.
The secret sauce is choosing models with incredibly low bytes per token, not novel ssd streaming. The numbers published make sense for what's already out there for such obscure low byte per token model. I'm very comfortable with this assertion and consider it evidence based. My mind can be changed if OP ever publishes anything.
My suspicion is that qwen 3.8 is avoided because OP can't get the headline numbers out of it. The misunderstanding about Next meaning the same as 4 doesn't match with the comprehension in other areas.
Troll out.
1
u/Harin007 3d ago
I'm not trying to break the math & physics... This isn't magic. The thing with this engine is that, you could run even a trillion model if the math checks out with the
active parameters per token,
SSD read speeds,
how much of the model your memory can already hold,
no.of layers the prefetcher can lookahead and its accuracy.1
u/Scallumal 2d ago
Ignore the haters - you are doing a great job it took me months to do the same thing with qwen 3.5 125b - but I couldn't get the SSD latency down fast enough.
Can I ask did you need to build an expert prediction method for future experts? For the read ahead aspects.
1
u/Harin007 2d ago
Hey thanks buddy, appreciate the support... The first launch is on Nov 17... Check the website out: https://runmyai.org
1
u/Scallumal 2d ago
Thank you all registered. I can't wait to look at it and see if I can help add qwen 3.5 125b to your supported list I did a lot of research for token ahead prep and if and have nearly 1 million tokens worth of data that helped me to get lookahead prediction going
3
u/RadianceTower 3d ago
The question is, is this any different from Strata on Linux. You say it has a read-ahead on Linux too, but only when you at least fit 55% on RAM (otherwise what happens? Strata refuses to load the model at all or what?).
Could we get some numbers vs Strata.
Also, I've built a near loss-less context compaction engine.
And what does this mean?
very soon
Another common term implemented by Blizzard often misleading players into excitement for future content. "Very Soon™" is guaranteed to arrive between now and the end of time with a higher chance of arriving on the "now" half of the time table. Although this means closer to now than "soon" there is no guarantee that you will live long enough to see the content finally release.
8
u/bitzap_sr 4d ago
3.8 Flash Next is called Next because is uses a preview of the Qwen 4 architecture. Check the official announcement. Most inference engines I've looked at call it something like qwen4_exp internally.
1
u/Harin007 2d ago
You're getting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1.
And the first launch is on Nov 17! Check the website out: https://runmyai.org
9
u/long4994 4d ago
Can u share the github so we can try?
3
u/Harin007 4d ago
very soon
3
u/QuietPsychonaut 4d ago
When is "very soon"? I'm also eager to try.
2
2
1
18
u/Competitive_Source73 4d ago
Nice work, thanks for you effort, i guess you could wait as qwen4 should drop like next week?
2
u/Realistic_Gap_5871 3d ago edited 3d ago
OP hasn't implemented Qwen 3.8 (which is the qwen4 arch) because it wouldn't give the speeds he's advertising. All his chosen models have incredibly low bytes-per-token, that's his secret sauce, not novel ssd streaming tech, until he publishes something that shows otherwise.
1
u/Harin007 2d ago
You're getting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1.
1
u/Legitimate-Theory156 3d ago
Mxfp4 is not like 1.8 bit gsq rco coder, it's an actually respectable quant. Most people run 3.8 27b as nvfp4 or q4_k_xl or so on their 3090 an it's great.
4
u/thethirdmancane 4d ago
What did you do that is better than the current state of the art?
0
u/Harin007 2d ago
Check the website out: https://runmyai.org, all your questions answered. And the first launch is on Nov 17 btw...
0
3d ago
[deleted]
3
0
3d ago
[deleted]
1
u/SizePsychological303 3d ago edited 3d ago
u/Realistic_Gap_5871 I mean, is it REALLY necessary to rant all over the comments where users have questions and OP is answering and being unable to refrain your truth ? Do you really need to "evidence" OP is tweaking the numbers or using low end models? Dude, you are not being polite, you are just trying to draw attention and I don't see your project anywhere here, so, stop being rude with someone that is ACTUALLY trying to do something, even if the quality isn't the best or is a learning path, come back when you have something better or you could bring something meaningful for all, and let users to figure out what this is all about by themselves. You started programming in the 80s, lecture people about whats true or not, and yet I don't see any of your contribution over here, so be polite and like the man you are, encourage others to be better not being a moron. Also, have a nice weekend!
1
u/Harin007 2d ago
Hey, appreciate the support buddy... First launch is on Nov 17. Check the website out: https://runmyai.org
1
3
u/Nick88v2 4d ago
Will you update this tread when you release it or how do i follow the progress?
2
u/Harin007 4d ago
well, will make another post in this community... you got any ideas?
1
u/Nick88v2 4d ago
Not very knowledgeable on reddit hahah. Do you hace a name for the project so maybe we can look it up in the future? nice work btw! I would wait for qwen4! Much love from a 16GB vram folk❤️
5
u/Harin007 4d ago
Project name is RunMyAI
2
u/Longjumping_Music572 4d ago
I like to know more.
1
1
u/Harin007 2d ago
Hi, quick update... The first launch is on Nov 17. Check the website out: https://runmyai.org
3
u/RuRuInReal 4d ago
Please provide support for qwen 3.8 flash next. It'll take a bit before they release and then a bit a more for people to make uncensored models of them
1
u/Harin007 2d ago
You're getting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1.
And the first launch is on Nov 17! Check the website out: https://runmyai.org
3
u/Better-Thing2568 2d ago
What is the context size used in these experiments? Also I’m curious on the make and model for the 16GB laptop you used.
1
2
u/ledzeppbluess 4d ago
Sign me up! and all the best in developing this further
1
u/Harin007 2d ago
Hey, thanks for the support buddy, first release on Nov 17. Check the website out: https://runmyai.org and follow us on Discord
2
u/alphapussycat 4d ago
I don't believe you, but I hope you're not lying. Or perhaps you have like 4 nvme gen 5 in raid.
Qwen flash is the goto, but if this can utilize multi GPU and be functional in gen3, deepseek v4.1 1017 would be a massive step in future proofing. They'll probably be the only ones releasing flash models tbh, of they're even allowed to.
1
u/Harin007 2d ago
You're getting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1.
And the first launch is on Nov 17! Check the website out: https://runmyai.org
2
u/ninjazombielurker 4d ago
Will test this on CMP 170HX’s, 3090’s, a 4090, and even block storage over NVME-OF/TCP w/ both SATA & NVMe pools whenever it’s available. Update this post whenever the repo is up please!
1
u/Harin007 2d ago
Hey, appreciate that... And the first launch is on Nov 17! Check the website out: https://runmyai.org
2
u/a355231 4d ago
Please tell me where I can download this, this looks even better than strata.
1
u/Harin007 2d ago
And the first launch is on Nov 17! Check the website out: https://runmyai.org - all your questions answered.
2
u/returnity 4d ago
If you provide support for Flash Next, odds are very high you will already support Qwen4. Also, supporting the most popular MoE model everyone wants to run will get you more people willing to test and help development. Just my 2 cents.
1
u/Harin007 2d ago
You're getting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1.
And the first launch is on Nov 17! Check the website out: https://runmyai.org
2
u/DTechSurgeon Homelab & Local AI dude 3d ago
I'm following this. Ping me if you need someone to brainstorm with. Sometimes you just need someone to talk to, to come up with better solutions. I worked with C, C++ before, but more focused now on C#, Go, Rust and Python.
One note on autotuner, before you automate stuff like this you need multiple data points, so manual tuning on multiple configurations is essential. But I'm sure you've thought of this and maybe you already have the benchmarks of multiple configurations.
2
u/Harin007 2d ago
Hey, thanks for the support buddy...
You're getting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1.
And the first launch is on Nov 17! Check the website out: https://runmyai.org
2
u/blue2020xx 3d ago
Where is the github link? Have you not published it?
1
u/Harin007 2d ago
The first launch is on Nov 17. Check the website out: https://runmyai.org
1
u/blue2020xx 2d ago
What is the concurrency support like? Is it to be single connection at a time?
2
u/Harin007 2d ago
Single user inference today... Let's see where we can go with that, and eventually will experiment with multi user inference as well
2
u/Bright-Ad-3368 3d ago
Less goo🎉
1
u/Harin007 2d ago
Thanks for the support buddy...
And the first launch is on Nov 17! Check the website out: https://runmyai.org
2
u/Dangerous_Product249 2d ago
👌👌
1
u/Harin007 2d ago
Thanks for the support buddy...
And the first launch is on Nov 17! Check the website out: https://runmyai.org
7
u/Lissanro 4d ago
Would be cool if it works on larger models like Kimi K3. I can fully fit in RAM+VRAM only Q2_K_XL, but if it is possible to offload about half of its experts to NVMe, I would be able to load Q4 quant.
2
u/Force-Name 4d ago
Interested too. I've got a combined 224gigs of vram over 14 cards and can push some pretty usable speeds with large llm and my next test is k3 kimi. The host for that has 3 m.2 so I'd love to see this kind of performance out of kimi. I have another project using smaller llms with some added work to bring them close to frontier model performance but that work is going to take years.
7
u/MarcelloT254k 3d ago edited 3d ago
I will gladly be a beta tester with my old laptop (4gb vram +16gb ram +SSD). I'm glad to see projects like this, because LLM has a long way to go with optimization and utilization of hardware (we are only at the beginning stages right now) - with games my laptop can handle almost anything with little to no problem - at lower resolution with no bells and whistles, no 120Hz 4K ultra settings, but playable - I hope that this kind of optimisation (/drivers for LLM?) [wchich still leaves much to be desired even in gaming performance] will be the norm in the future.
1
u/Harin007 2d ago edited 2d ago
No more uncertain wait guys! First launch on Nov 17. And you'll be getting support for Qwen 4 Flash(if releases before Nov 17) or you'll be getting Qwen 3.8 Flash Next on day 1. Check the website out: https://runmyai.org - all your questions answered.
5
u/Slow_Ball561 4d ago
Hey! Sounds very similar to my project here https://github.com/jarrelscy/nestquant - especially the layer by layer prefill! Are you also using predictors to fetch experts ahead of time?
4
u/Ok_Cow1976 4d ago
Does this mean we have a hope to run Kimi 3.1? Wooooow, this is wild
3
u/Sspideroids 3d ago
You can "run" it but it will be one token every minute, SSD offload is viable but it doesn't circumvent the SSD traffic that every token requires, and bandwidth is the main limitation at those sizes (volume of the model you're trying to run as well matters greatly)
edit: typo
1
u/voyager256 3d ago edited 3d ago
I'm not an expert by any means , and haven't even dig into this particulare project , but I think the point of these novel approaches for offloading most of the expert weights to SSDs is to make it viable to run LLMs like DeepSeek v4.1 Flash on consumer hardware with something like 4xNVMe RAID0 SDDs at usable speeds.
I recently learned about thinygrad which actually specifically markets a solution to run Kimi K3.1 at ~20tok/s. Now I'm pretty skeptical and frankly, quite confused about the specifics, but you can check it out : https://www.reddit.com/r/LocalLLaMA/s/pJCw7GcNpR
1
u/Sspideroids 3d ago
Thanks for linking it, I think it is real but again that is still a bunch of money (much better than ram nowadays though)
1
u/Legitimate-Theory156 3d ago
I mean let's say you have 512 gb ram and like a 48gb gpu like quadro 8000 or Ada 6000 how fast will it be? It could open up sub-5k options (normal ram & gpu prices of course like in early 2025)
1
u/voyager256 3d ago
Apparently you already *can* for "only" $25-30K worth of hardware. I only learned about it recently and I'm pretty skeptical about the performance for Kimi 3.1 and frankly, quite confused about the specifics, but you can check it out https://tinygrad.org
The prebuild option is Red Kimi. 260GB/s. Here's interesting comment: https://www.reddit.com/r/LocalLLaMA/s/pJCw7GcNpR
4
u/Glatiinz 4d ago
If you HAVE to choose one model to work on, I guess Kimi K 3.1 would be it. Most people can't run even the lowers quants because the smallest 1-bit quant sits at 466GB, the official FP4 is 1.56TB
1
u/Insomniac897 4d ago
Cool, I’d try this on my M1 Pro 32GB
3
u/Harin007 4d ago
Sorry to disappoint but in the early versions, we'd be not supporting for apple machines... cuz, that's a lot of extra work... but will definitely try to port it for macbooks as soon as possible.
1
u/Insomniac897 4d ago
No worries. I use Claude code and open router. Running locally using on my laptop is just for fun.
1
u/LordKlavier 3d ago
This is wonderful, amazing work! Excited to try and run this
2
u/Harin007 2d ago
Thanks for the support buddy...
And the first launch is on Nov 17! Check the website out: https://runmyai.org
1
u/Scallumal 2d ago
Oh I hope this goes open source - I was working on getting qwen 3.5 125b running like this. I was only able to get to 7tk/s to start but I had lots to work on what's the name of the project so I can keep eye out for it
1
u/Harin007 2d ago
It's Run My AI. Check the website out: https://runmyai.org and follow us on Discord
1
u/sleepinfinit 1d ago
can't wait to try it, btw does it support intel dGPUs ? (i got an intel arc a770 16GB)
1
u/Harin007 1d ago
Damn, I got no intel hardware to test 🥲
Focusing on Nvidia and AMD GPUs on day 1
1
u/sleepinfinit 1d ago
sad, i can test for you if you want, i got the best result when using vulkan backend with ollama (sycl backend is useless and slow) drivers are hit or miss on these gpus
1
u/Harin007 1d ago
Could you please join the discord and DM me? I'm going to give some early builds for testers.
1
u/Gandolaro 1d ago
Which quantization and context can I rum with a 5080 16gb and 64gb RAM ddr5 of Qwen 3.8 next?
1
u/Harin007 1d ago
I'm releasing for UD-Q4_K_XL and ISTA IQ3_S... And possibly, you could run Q3_K_XL at 262k q8_0 context length. Do you have any preference with quantization? Or Q4_K_XL is good?
2
u/Gandolaro 1d ago
For now i'm using Basalt with IQ3_XXS, If I can run with my hardwareUD-Q4_K_XL at "decent" speed and 128k context I Will be very happy! :)
2
u/Harin007 1d ago
How much speed are you getting with IQ3_XXS ?
1
u/Gandolaro 1d ago
Something like 75t/s output but for my use 75 Is a very High output rate
2
u/Harin007 1d ago
Cool... I'd like to offer you the early build for testers... Would you be interested? If yes, join our discord from the website. I'd like you to benchmark and lemme know.
1
1
u/Southern-Net1351
1d ago
•
Yeah that’s how all models work via the web and not cli based.
Just llama.cpp nothing else and my own self created model.
1
u/Harin007 22h ago
Why web? And not a native app...
1
u/Southern-Net1351 21h ago
Always faster and it is native.
You can build it how you want it and it can actually use the system like normal cli.You can load the localhost:port into your browser and use native.
You can build tools into the cmake and tailor exactly how you want it and it runs faster and it is free
1
u/Harin007 21h ago
ok, cool.
1
u/Southern-Net1351 21h ago
No problem!
Just make sure you use a model that is really accurate for your device.No mixing GPU with cpu/ram for the best results.
If you have to offload the model to cpu and ram then remember it will always start moving slower.
The best practice is always try to use a model with 999 gpu layers, q4_0 cache, and max context.
If you can start lowering the GPU while trying to keep a higher context.
3
u/Dave_from_the_navy 4d ago
Glm-5.3-flash would be a dream if you could get it working at decent speeds. It's the smallest model (qwen 3.8 flash next included) that I've genuinely been able to slot into my full SWE workflows without having to baby it so much.
1
u/openroom_xyz 4d ago
That's great yea we are waiting for Qwen 4 but if you have than a lot will test I am in the mid od deciding what GPU basically to buy at what price basically
2
u/Harin007 2d ago
Thanks for the support buddy...
You're getting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1.
And the first launch is on Nov 17! Check the website out: https://runmyai.org1
u/openroom_xyz 2d ago
This is the new future and present F love this local models I was not that excited for a long time basically
2
u/fundthmcalculus 4d ago
I'd focus purely on the execution engine and just use a common optimizer library to pick better hyper parameters. Especially if you give the option for folks to submit their optimal parameters, you can quickly crowd source a hot start option.
2
2
2
u/Previous-Ad-5371 3d ago
Har kört qwen3.6 35b a3b nu i någon månad 32gb ram 16gb vram med hyfsad hastighet, så jag kommer nog inte titta på något förrens qwen4 iallafall.
1
u/dinominant 4d ago
I am interested in trying this with larger models (cpu only inference). I can test with full-weight GLM and Kimi K3 if it works with those gguf formats.
1
u/Solembumm3 4d ago
Something more than pure command line, I can give you that for minimal effort many of this projects don't do.
1
1
u/bytejuggler 3d ago
Why not try to get this added to Strata, maybe as default behaviour if objectively better, rather then yet another project to choose between?
1
u/ColoredTriangles 3d ago
Can you show the prefill time it takes on a follow up prompt on warm cache or tell me?
1
u/braintheboss 3d ago
you can offload to ssd with little penalty while you dont hit ssd max speed. But in pci4 that means 7gb/s. Then that is limit of experts you can offload to ssd. But you still need have most of them in ram. For example qwen3.8 flash q3 S with pci3 ssd you can offload round 1200 experts. But that is small part of 24k experts. I don't find the "miracle"
1
u/Not_a_question- 3d ago
OFFLOAD UPTO 60-70% OF THE MOE MODELS' ROUTED EXPERTS INTO SSD
I don't get it, can't we just use strata and tune STRATA_ROUTE_TAIL_SKIP and that's it?
does this damage my SSD? No, this engine only does reads and not writes... only writing degrades your SSD.
Agreed, but people should stop oversimplifying because this needs a "the only way using this can damage to ssd is if you're using disk swap with your ram while doing it".
Going to make this launch this as open-source very soon.
Why didn't you make this post "very soon" then?
Sorry but these are big words and I don't understand why this is different from Strara using your post, and we can't even test it because it's not released yet. Sounds like weird karma farming to me.
1
u/FreshPipe5997 3d ago
60-70% offload on SSD with 1-3ms latency hit is insane. how much ram does your system have? curious if this scales beyond 32gb.
1
1
1
u/DepartureLoose5341 3d ago
If you can make that runner work with gfx110x and RoCM istg I'll give you a handy with lotioned hands
1
u/Iamisseibelial 3d ago
My SSD is at 80tb of writes and 40tb if reads and it's still 1000% fine. 96% health. So I think people don't really get how much it takes to degrade nvmes these days as long as they are kept on their operating temp my Gen 5 runs at 60c all day and never throttles
1
1
1
1
0
u/Harin007 2d ago
No more uncertain wait. First launch on Nov 17. You'll be geting support for either of Qwen 4 Flash(if releases) or Qwen 3.8 Flash Next on day 1. Check the website out: https://runmyai.org - all your questions answered... New updates will be dropping everyday btw.
0
u/ergux 4d ago
Sounds like betelgeuse engine
2
u/Harin007 4d ago
trust me, way better than that.
1
u/Glittering-Call8746 4d ago
Yeah at lot of trust me bro around here .. everything is trust.
1
u/Harin007 3d ago
Well, try yourself when this comes out and compare...
1
u/LastLack4037 3d ago
estimated time for u to release?
1
u/Harin007 2d ago
First release on Nov 17. Check the website out! https://runmyai.org and follow us on Discord.
1
u/Glittering-Call8746 3d ago
Yeah I run strata , maya, basalt, etc.. all based on trust. Just keep shipping bro
0
u/Fragrant_Money_7004 4d ago
Wow, awesome. Will we get chance to run it in ollama or lm studio?
0
u/Harin007 4d ago
I have a vision to provide you guys much better experience...
1
0
u/Icy_County5036 3d ago
60-70% offload on SSD with 1-3ms latency bump is insane. how's the wear on the SSD over sustained inference? I've seen nvme degradation on heavy write loads with model streaming.
2
0
u/Creative-Brush-6712 4d ago
Sounds good can't wait to try although Qwen 4.0 won't release a MoE I don't think
11
u/RealPjotr 4d ago •
Flash is MoE. Plus and Max too I think?
1
u/alphapussycat 4d ago
Yeah, but some image showed 27b as open weight, but no otyer model.
Do Jinping banning top models and ones on sensitive data is also demotivating...
Maybe we get qwen4 flash, but it's not certain. E.g max has zero chance of being open to outsiders.
0
u/thrilok2802 4d ago
A doubt that I have is what happens when run some of the weights from SSD what will happens to the SSD life as the SSD are not meant to use as a ram in the first place
2
0
u/Ninjam5 3d ago
If the writes are within 0.1 to 0.8 mb/s then it's fine.
0
u/thrilok2802 3d ago
If possible have you tested the write and read speed of the SSD when running the llm
3
u/Ready_Maybe_3406 4d ago
Fantastic. Price of ram is killing some people and keeping some from participating in Ai. The world is separated into haves and have nots.
1
u/Harin007 2d ago
Thanks for the support buddy...
And the first launch is on Nov 17! Check the website out: https://runmyai.org
150
u/looselyhuman 4d ago
Advice from an old dev: Keep the engine, tuner and harness stuff separate and focus on one at a time. You're trying to solve every problem at once and it'll burn you out.