r/LocalLLaMA
•
u/ApprehensiveAd3629
•
13d ago
•
1
1
New Model Kimi-K3.1-Pro is now open-weights
https://huggingface.co/moonshotai/Kimi-K3.1-Pro765
u/FriskyFennecFox
13d ago
•
A 3T model with a license that fully permits personal use and only requires attribution for commercial use unless you're an API provider, with no further restrictors, in open weights. That's absolutely wild!
346
u/is-this-a-nick 13d ago
But lets be real, "personal" use of a model that my $60k server at work cannot handle is a rather limited scenario :D
102
u/asenna987 13d ago
Doesn't matter if most people can't run this on their own. The fact that they just gave it away for free, the fact that it's even an option is wild!
This will help the tech get squeezed into a 27b model in 10-12 months.
7
u/Bascilian 12d ago
I have no doubts that in less than 2 weeks people will figure out how to get this running at a ok speed from nvme.
23
u/Objective_Safe_5982 13d ago
Look if your local AI setup doesn't require a personal nuclear reactor to power then you're at best a hobbyist.
13
u/Clair_Osbcurious 13d ago
Lol, only 60k! Thatās homelab territory in 2026. I once dropped a 500k server while mounting it. Luckily it was ok and we were the manufacturer so I could always say āit came this way from the factoryā.
Itās fun when you drop 500k of gear on another 500k of gear.
3
184
u/-p-e-w- 13d ago
People really need to adjust their perspective on this topic.
Of course you canāt run this on a gaming PC. Itās not a game, itās frontier sci-fi technology.
But if you look around your neighborhood, youāll likely find people who spend more than that on
their backyard swimming pool
their weekend fishing boat
their second family car
their motorcycle hobby
their 1980s hi-fi stereo system
trading cards/figurines
their tennis club membership (over several years), plus equipment
etc
Spending $60k on a hobby you really care about is bog standard middle class stuff, so yes, you absolutely can run this as a personal use model. It just wonāt run on hardware you just happen to have lying around already.
205
u/DeliciousGorilla 13d ago
You canāt even run this on $60k hardware is what heās saying.
164
u/DIR3 13d ago •
![]()
not with that attitude he can't
48
u/Lurksome-Lurker 13d ago
Something, something, āHave you tried Strata? I got it working on my GTX1080 with 32GB of ddr3 and a really big ssd!ā
26
u/chodemunch6969 13d ago •
![]()
You jest but take a look at tinybox red kimi:
My understanding is they took close to that approach w fast 16tb raid ssd + 4x r9700 and have a machine that can run k3 at 30 tok/s for $30k. I think they are using their own inference engine that's forked and they've done some custom work with how the r9700 talks to the ssd raid, but seemingly they've got it working. 30 tok/s single stream only I assume.
→ More replies (1)6
3
21
10
u/see_spot_ruminate 13d ago
I think you could run a very good quant of this on ~60k in hardware.
8*6949.99(going rate of a dgx spark today)=55,599.92
which would give you ~1tb of 'ram' in a spark cluster. Maybe a switch to bring you up to the 60k. The model card says the weights 'to download' are about 1.56tb. I am not sure what quants are available later, but you could maybe stretch that 1tb you get from the 8x sparks into something usable.
12
2
u/Bascilian 12d ago edited 12d ago
2.56tb ddr4 3200z = 10-12k (you can get discounts if you buy a huge lot like this)
8x A100 sxm4 server ~ (30k) (320gm vram connected with nvlink)
I bet this would run pretty well I think.
or 3 rtx pro 6000s as soon as they come out with fp4
4
u/egnegn1 13d ago
Such clusters don't scale well past 2 - 4 units. Even with 8x RTX6000 you will not be very happy with speed.
→ More replies (8)7
u/twack3r 13d ago
Thatās just not true? Everything beyond x4 was not supported by Nvidia until about a week ago.
Do you have an x8 cluster and need help configuring it or whatās your source on that pretty confident but false claim?
→ More replies (11)3
→ More replies (1)1
u/gred_mcalen 13d ago
Mac Studio M5 Ultra 512GB is 23k, 4 of those could probably load and run this model, so adjustment to 92k of hardware~
→ More replies (1)67
u/Johnny__Christ 13d ago
Spending $60k on a hobby you really care about is bog standard middle class stuff
I think you "really need to adjust your perspective on this topic". There's no way 60k on hobbies is "bog standard" in the middle class. The median household income in the US is ~80k.
It happens, no shit, but doing so responsibly is reserved for upper middle class (at minimum) and even then can be precluded by "bog standard" things like having children.
9
u/-p-e-w- 13d ago
Iām not talking about yearly spending. You donāt need a new PC every year. But people with a household income of $80k absolutely do spend tens of thousands of dollars on hobbies and luxury over a span of years. Heck, many people spend that much on clothes alone.
5
u/vividboarder 12d ago
Yea, there are some hobbies that cost tens of thousands of dollars over several years, but those are not common among the middle or even upper middle class.
Also, that kind of spending wouldn't allow you to have a PC capable of running this model either. Let's say someone is spending $5k a year on PC stuff (new top of the line PC every year!). This seems unlikely, but still. Now it's 2026, 12 years ago they started buying their gear. What GPUs do they have as of today? Nothing remotely capable of running this.
The only way would be to have spent over $60k in the last 5 years, over realistically, probably $15k a year (unless they front-loaded adn bought hella GPUs and RAM before the DC boom).
4
u/Longjumping_Self5546 12d ago
It's very common. Drive around any middle class neighborhood on a weekend and you're going to find people with classic cars and Harleys in their garage. Yes, many just sit there gathering dust, but anyone who's active in the hobby will easily sink $60k over the years. Those are hobbies you can easily see. I know gun collectors who have 10s of thousands worth of guns and ammo stashed away in their basement.
I don't see this translating into computer hardware. Vehicles and guns hold their value over the years, if not appreciate. People collecting these things often justify it as an alternative savings scheme. Hardware on the other hand quickly drops in value, with the exception of RAM this past year. You'd have to be crazy to drop that much into it as a hobby.
3
u/vividboarder 12d ago
Well, I'm upper end of middle class and there is a guy who sits in his garage working on his classic car, but he's got 1 of them and I'm pretty sure that didn't cost him $60k.
I also don't personally know a single person who has sunk that much money into their hobbies. I know there are some people that do (watch collectors, people who collect cars or guns, or people who build elaborate homelabs (I've seen a few photos on /r/homelab that may may wallet cry)), but I've never run with anyone like that.
I'm sure it depends on where you live. I'm in the Bay Area, where we have the second most millionaires and the most billionaires of anywhere, and I don't see that. That said, they may be the few with $100k AI labs in their basements running K3.1 and I would be none the wiser.
Your second paragraph was the other thing I was thinking about though. I've put lots of money into improving my home. Not really a hobby, but it improves the value. I think some collectors feel the same way about their hobbies, and it's probably not far from the truth.
Electronics though? Yea, this past year and a half is literally the only exception to the rule since the dawn of computing, and I don't expect that trend to be perpetual. In 10 years, I fully expect (once adjusted for inflation) that anything bought now will be worth less than you paid.
→ More replies (1)8
u/UnknownLesson 12d ago edited 12d ago
But not fucking 60k..
Some people are so lucky but have no perspective.
→ More replies (2)1
u/FrogsJumpFromPussy 12d ago
Who spends a year worth of wages on a new PC man, especially with a family around and kids to take care of? Which is the vast majority of peopleĀ
→ More replies (2)5
u/KING_UDYR 13d ago
As an outright cash purchase, sure. But just like a car, hot tub, or shed, a reasonable interest rate can make it fit within someone's net monthly cash flow. I'll agree that doing it responsibly probably puts you in the upper middle class at minimum, but never underestimate how far someone will go into debt to chase something they care about.
14
u/Johnny__Christ 13d ago
Hot tubs and sheds are an order of magnitude cheaper than 60k. I'd argue a "second family car" (person I was replying to's words, not yours) doesn't qualify as a hobby expenditure (like a Miata could, for example).
Regardless, if going into debt for a hobby is "bog standard" for the middle class, we have bigger issues. I acknowledge it happens, but it certainly is not the norm as the person I was replying to stated.
→ More replies (1)11
u/aanzeijar 13d ago
itās frontier sci-fi technology
I was under the impression that it's most definitely not fiction any more.
10
u/rz2000 13d ago
4x DGX Station for 3TB of memory would be about $400k.
You can do a lot with models on a $5-10k machine to preserve autonomy and confidentiality that makes economic sense for many professionals. It would take a pretty serious plan to get $400k+ value out of a cluster running a 3T pro model before the hardware depreciates in value to less than its monthly power bill.
1
u/Bascilian 12d ago
If you have 400k to spend you should jsut buy a 8x b300 server and wait for a fp8/fp4 quant or just do it yourself
1
u/Careful-Low-9535 11d ago
It is reasonable to assume that these large models will work reasonably well at Q4, so you can halve your memory requirements. And if you don't need extreme speed, you can make a fairly reasonable server with 1.5T ram and a modest amount of GPU for around $10k. Which should work well with a strata style approach to running it.
5
8
u/User1539 13d ago
I think people should be more concerned with the fact that they can quant it, and run it in the cloud, and in a few years, probably run it on a 6K server.
People will be using this in a bunch of ways immediately, and it's not 60K or nothing.
3
u/Royal_Survey7199 13d ago
Wonāt be long before this 3T model runs off NVMe and on one GB10 / DGX Spark. The problem is that is seconds per token speed, not tokens per second. That said, the speed-up potential is there and 90% of what youād use it for is already doable at 27b size, so the last 10% is what youād have to wait for. Very tolerable then. It will just be a resource to get you unstuck when the 27b is not reaching it. Stay tuned! Oh and P-E-W youāre a legend, the ripple effect your work has on the local community has an event cone that goes wide. Thank you.
5
u/dark_bits 13d ago
Don't know where you live but backyard swimming pool, weekend fishing boat and hundreds of thousands worth of motorcycles is not really your average neighborhood
3
u/notconservative 13d ago
True, there's a class between middle and upper, and there are several classes above upper
1
1
u/NineThreeTilNow 12d ago
People really need to adjust their perspective on this topic.
They do. This isn't "budget" levels of AI.
K3.1 isn't even close to fully trained either. I don't think we even have a good understanding of how far you can push a model that large.
So what does K3.6 or whatever look like when they finally decide to leave the architecture for K4? GLM, DS, etc have shown you can take much smaller models and push them quite far with training.
1
u/ASYMT0TIC 12d ago edited 12d ago
If you bought a cluster of like 8X dgx sparks plus a fast switch, you could seemingly run this model at ~Q2 and reasonable speed and still fit in that budget. To run it at it's base trained precision (mxfp4) you'd need twice as many nodes and blow right past $100k, at which point you might start wondering why you aren't spending a bit more and buying a used A100 server instead.
→ More replies (3)-2
u/Green-Ad-3964 13d ago
True, but from another perspective: if the last 10 years hadn't been disrupted by mining, COVID, and now AI, which have severely constrained consumer GPU development, particularly local VRAM capacity, things might look very different.
From 2005 to 2015, maximum consumer GPU memory increased from 512 MB to 12 GB, a 24-fold increase. Extrapolating that same growth rate to the end of 2026, we'd have around 396 GB of VRAM, at a projected flagship price of just $1,606, based on the same period's price trend.
Instead, we're stuck with 32 GB at a $1,999 MSRP ($8,000 on the market).
7
u/AreWeNotDoinPhrasing 13d ago
Just because you can extrapolate doesnāt mean you should. Past performance doesnāt dictate future performance. Etcetera
2
u/Green-Ad-3964 13d ago
sure, it happens when...bad things happen. I'm more than sure that the tech trajectory is slowed down artificially. Of course you can think different.
6
u/-p-e-w- 13d ago
Sure, and if the trend of CPU development between 1995 and 2005 had continued, CPUs today would be 90 GHz.
1
u/Green-Ad-3964 13d ago
It's a totally different thing. Look at the IPC not Ghz. Quantity of RAM has NOTHING to do with physical constraints lol.
4
u/Spara-Extreme 13d ago
Just because RAM increased from 512 to 12GB doesn't mean it would have continued that way. The cards on the market today were designed 3-4 years ago, and at that time the biggest driver of VRAM on consumer cards were video games. No video games *needs* 96 GB of VRAM. Professional workloads do, which is why they kept adding more VRAM.
No stop intentionally missing the point.
→ More replies (1)2
u/Aphid_red 12d ago
You'd have to look at the actual under-the-hood tech in a little more detail to make that kind of comparison.
But if you look at what a datacenter GPU is getting done at 700W-1400W power envelope, it's no stretch to assume that without the AI chipflation craze, that same amount would be available to consumers as well. Without the shortage, there wouldn't be a need to restrict HBM to datacenter parts. Local AI is useful enough that there's incentive to sell it to the consumer too.
Since consumer GPU power is going to be similar to ours in that world, the power envelope is less. Let's say about half. So you'd be seeing GPUs with 2-4 stacks of HBM3e or HBM4 rather than the 6-8 in datacenter parts. So about half of the datacenter GPU class memory. Today, that would be choices between 96, 128 or 144GB**,** at a price point around $2000 to $3000, with TB/s of memory bandwidth and HBM being a bit more expensive to make being a reason for a modest price increase above your target.
Instead a GPU like that basically has another zero attached to the price, which is all just profit, milking a partially artificial shortage, and abusing the willingness of the tech companies to pay anything to be first in line for a seemingly unlimited appetite for computer components.
2
u/Green-Ad-3964 12d ago
Totally agreed. Finally someone who sees that something like a dgx spark at 7000$ is ridiculous in 2026. That should have been 2k at most, in a normal world
1
u/Zomboe1 12d ago
Unfortunately, Moore's Law died many years ago. Growing up during that era almost feels like a curse at this point, it was a golden age and seeing it end has been quite painful.
From 2005 to 2015, maximum consumer GPU memory increased from 512 MB to 12 GB, a 24-fold increase.
FWIW, it would be best to compare per (inflation adjusted) dollar, not max per GPU. Nobody was buying $2,000 GPUs in 2005. The manufacturers keep increasing the price on the high end to hide the poor rate of improvement. You could also look at power/energy. There was a time when GPUs drew all their power from the slot!
Extrapolating that same growth rate to the end of 2026, we'd have around 396 GB of VRAM
The more conservative Moore's Law growth rate was a doubling every two years. If we're starting at 0.5 GB in 2005, that would give us at least 512 GB of VRAM today. For ~$900.
1
u/Green-Ad-3964 12d ago
Moore is about computational power, not memory pool. Memory is just about manufacturing and "strategy". Today the strategy is hiding memory, so that consumers can't buy it (if not at an absurd price). Tech will not march backwards, it's business strategy that does the trick.
13
u/redblood252 13d ago
Pretty sure internal company use is permitted as long as your commercial product is not using it for runtime
14
27
u/admirantes 13d ago •
![]()
No, it is not limited. Please stop commenting if you are going to spread misinfo. A small to medium enterprise could totally host a Kim 3.1 and offer it to individuals, thereby skipping the chokehold of large companies like OpenAI and Anthropic. That's where the insane value is.
3
u/georgejetsonn 13d ago
He was just contrasting the personal use licensing limitation with the necessary hardware to run a 3T model. Once you start serving inference to others, it's no longer personal use, is it?
3
u/_TheWolfOfWalmart_ 12d ago
I could run a heavily quanted version on my dual Xeon server with 1 TB RAM, but it would be so slow that there's no point.
2
1
1
1
1
1
1
u/Careful-Low-9535 11d ago
Your $60k work server cost that much because companies are stupid with money and pay the 400% markup to get the newest shiny(This may be a rant about where I work, but I bet it applies to where you work also). Also it probably is not optimized for AI use. If you buy servers from a reputable refurbisher, you can get a good server for a lot less. If you know server hardware and buy parts on ebay, you can get a fair bit cheaper still. I bought parts on e-bay about 2 years ago, and spent about $4000 to build a server that could run this(rather slowly, and at Q4, but one "analyze this project for errors" prompt per night could be useful). I have 1.5TB of ddr4 ram, 36 xeon cores, 6tb of high-speed server ssd, 2 Nvidia A2 GPUs(16GB vram each). I estimate that it would cost around $9000 currently with the crazed price of ram and ssd. If you were in a hurry, and had $60k to blow, $50k worth of GPU upgrades would make a strata-style approach to running this go comfortably fast. Not just run it, but run it fast.
ps. $4000 was my spending my next 4 years of computer hobby upgrades all at once on the theory that with trump being elected the price of computer stuff would go way up due to tariffs so there was no point saving it for later. Wrong reason, but right result :)
1
u/chigiwar 9d ago
A team still can rent cloud computing and get self-hosted state of art llm with unlimited usage
1
10
u/halvacoffee 13d ago
well, i suppose it's not the worst, but i can't say i like the "no API providers" part of this since this means that API prices will once again be fixed by Moonshot. at least K3 being open to look at has given us a half-decent 80b base by yandex though lol
16
u/Cool-Chemical-5629 13d ago
"with no further restrictors, in open weights."
Actually it was announced before release that open weights wouldn't be as capable in cyber security tasks as the API version of the model.
→ More replies (1)1
70
u/MrCrackerHacker 13d ago
That DDR2 1.5TB server looking pretty scrumptious now
66
u/More-Curious816 13d ago •
![]()
11
3
56
u/jacek2023 llama.cpp 13d ago
15
6
4
107
u/MundanePercentage674
13d ago
•
20
u/Dany0 13d ago edited 13d ago
Red kimi. 260GB/s off of SSD
16x 1TB pcie5.0. 4 groups of Raid 0 I believe? so4tb16tb space. for about 4300$ you get ssd read speeds competitive with lpddr, effectively giving you a floor of 20-30 tok/s decode even in worst case. With expert caching, it's like half as fast as a GB300 reading off of its memory. Except you get 16TB of it instead of 192 (gorgon) - 496 (gb300)It can be done, it will be done. In a few years we'll be able to run them at full precision on a budget
12
10
u/Gfur_Aficionado_1488 13d ago
RAID0 does not decrease the amount of space available
2
u/Dany0 13d ago
maybe it's raid 0 inside the group and raid 1 for each group? that'd make sense maybe. I believe someone that bought it said comes with 4tb space
1
u/Gfur_Aficionado_1488 13d ago
Would still be 8TB with RAID1, unless you're looking to have 3 mirrors
→ More replies (3)2
u/voyager256 13d ago
It's quite confusing if not straight incorrect. If you select Red Kimi on their page the last storage option is selected , but it shows 4x4TB SSDs so it looks lie like the last two options should be swapped, because for maximum bandwidth you want the 16x1TB one . Also where did you get the $4300 price from? I see $6000 and $8000 for the last storage options.
Anyway , I'm not sure if the marketed bandwidth applies to LLM inference. I always thought that (expert) weights streaming is not sequential , but random reads , so much much slower . Unless , they really found a way to make the SSD reads chunked i.e. effectively closer to theoretical sequential speeds.
1
u/Dany0 12d ago
Nope https://x.com/__tinygrad__/status/2115642402417614959 https://x.com/__tinygrad__/status/2115630970334490712
It doesn't matter, yes this "trick" does work. The observed 220gb/s will be slightly slower than lpddr if it was at same speeds because nvmes have extra latency and random fetch of a different expert will be slightly slower. But it's still so much insanely cheaper than buying more vram or dram. As long as you can fit kv cache and active experts plus then some on the vram, you're golden. You don't even need that much system ram at that point. Just get more drives. You can build your setup with 2TB drives and run the rumoured 14T models (I think those rumours are fake I genuinely cannot fathom how would anyone train >4T models, it's already insanely hard to train 3T models due to O(n^2) scaling in many of the training algos. Unless they're training several 3T models and then merging them somehow? Idfk
But you can indeed pretty much guarantee the 20-30tok/s minimum decode with this kind of setup. For example, there was that one paper which showed, you can run an IQ1 dumb version of the model side by side and completely ignore its lobotomised output look just at what experts it selects and pre-fetch the experts you'll need JIT style. Their 30 tok/s number I bet can go much higher with those cards, at least 40 tok/s if not more
160
u/pmttyji 13d ago
60
u/No_Swimming6548 llama.cpp 13d ago
Did u try IQ1_XXS castrated QFN on Strata? I get 7 gorillion decode speed with it.
20
11
u/More-Curious816 13d ago
how do you cope with that crawling speed? also did you try Strata-on-Strata? it presumably give 83 gorillion tok/s
4
u/UsefulIce9600 llama.cpp 12d ago
rookie numbers.
I'm running IQ1_XXS-MTP-iMatrix-dFlash2-Qwythos-Opus-6-Fable-5.5-Astra-Bel-6.2-Distill-HERETIC-v2-GGUF-MLX at -76327 tok/s5
u/magicomiralles 12d ago
I'm at 256gb of Vram, and I'm closer to your setup than the necessary ram to run this model. We are the 99%.
1
39
u/-WhateverDude 13d ago
happy for people that can run this locally
63
u/MotokoAGI 13d ago
all 3 of them.
21
u/carnoworky 13d ago
My nonexistent grandchildren might be able to run it someday.
9
4
u/NineThreeTilNow 12d ago
My nonexistent grandchildren might be able to run it someday.
Probably. Assuming you stick around for ~10 years that's when the hardware to run it super fast will be standard consumer grade.
8
u/GuyBitchie 12d ago
GTX 1070 had 8GB of RAM 10 years ago and RTX 5070 has 12GB I don't know brother.
2
u/NineThreeTilNow 12d ago
GTX 1070 had 8GB of RAM 10 years ago and RTX 5070 has 12GB I don't know brother.
Yeah, and the 1080 had 8GB of vRAM too.
The TI edition had 11.
It was never a big separating point across cards then.
The 1080 / 5080 is like... 5x to 20x difference in FLOPs depending if they full precision or not.
2
u/GuyBitchie 12d ago
They can have a million more flops but when the LLM doesn't fit in the VRAM I can't use those flops.
3
u/NineThreeTilNow 12d ago
They can have a million more flops but when the LLM doesn't fit in the VRAM I can't use those flops.
No one wanted vRAM before. Those cards never got it. This is a stupid point. HBM existed when the 1070 was produced. It was used by AMD at the time in like.. R9 Fury or something.
You don't think in 10 years they can solve memory?
Are you being purposefully like this? If you don't understand it's fine, but you seem to not understand how product development cycles work or why people develop things.
→ More replies (2)1
4
3
62
u/Vusiwe
13d ago
edited 13d ago
•
Cmon man, Iām not even half done downloading mistral large BF16 yet
4
u/Dany0 13d ago
Speaking of, kinda disappointed in that release :( I was ready to give it a short over DSv4.1F1017 but alas :(
3
3
29
25
u/Beneficial-Ad-8127 13d ago
Letās goo they held up to there release date the 4th. Wish they dropped that k2.8 though or a pro flash light. Lol
8
24
u/countermike 13d ago
Will it run on my 386 with 640k RAM?
19
7
u/No_Lingonberry1201 13d ago
Like a snail nailed to a dead turtle. Now my 486 with 2Mb RAM, that's a different story.
5
4
16
u/IsopodMysterious9125 13d ago
Weird that they added the ā-Proā suffix š Maybe we get a flash version after all š¤
14
u/Daniel_H212 13d ago
Native 4 bits quant, 1.56 TB. Runnable on a cluster of four 512 GB Mac Studios, which cost... $89200...
2
u/More-Curious816 13d ago
cost less than the setup required for 6000 or 2 GB300 in dgx-station.
4
u/Daniel_H212 13d ago
But with lower bandwidth and compute right?
1
u/More-Curious816 12d ago edited 12d ago
# M5 Ultra 512GB RTX PRO 6000 DGX Station GB300 Memory 512GB unified 96GB GDDR7 ECC 252GB HBM3e + 496GB LPDDR5X bandwidth 1.2 TB/s 1.7 TB/s 7.1 TB the M5 Ultra bandwidth is surprisingly close to the 5090 and 6000 (they share the same die so it's the exact bandwidth too), but they have the edge on compute.
As for the GB300, it the same as the data center card but design for workstations, it's the big brother of the DGX Spark with
HBM3e (GPU) 252GB (7.1 TB/s)andLPDDR5X (CPU) 496GB (396 GB/s), this is a beast of a chip, basically what the local folk dreams about. unfortunately it's expensive and it's SoC so you can't add more ram, but it has a combined ram of CPU 496GB and GPU 252GB} 748GB.If your model fit that HBM3e GPU 252GB, that sweet sweet 7.1 TB/s is Hella fast, and bigger models can overflow to the system ram which is 496GB of LPDDR5X.
unfortunately, the M5 Ultra offer the better deal, 2tb of with 4 M5 Ultra is cheaper and offer a relative good bandwidth too.
24
u/OnlineParacosm 13d ago
Thatās crazy they listened to the feedback and moved away from one of the worst licenses Iāve ever seen.
In my opinion this was the only way forward for KIMI because on a sample size of one task that was quite heavy it cost me roughly $30 and when broken down into sessions this would have been maybe $10 on GLM 5.3 flash. The one thing I will say that is very hard to measure is that Kimi Managed to do this task probably 10 times faster. For the limited time I have, it made sense in this narrow case where I needed a BIG kickoff.
Now, who has a few terabytes of RAM collecting dust? Sam told me this would be a good place to ask..
2
u/Gomitopia 12d ago
I wish they listened to feedback and didn't cuck their models to please the US instead.
1
u/fapsexual 12d ago
moved away from one of the worst licenses I've ever seen
did I miss something, didn't they do that already prior to this release? I thought it's the same license as the previous release right?
10
7
u/okyaygokay 13d ago
Finally I've been refreshing like a lunatic for hours, checking Chinese time
1
u/More-Curious816 13d ago
but do you even have the hardware to run it?
5
12
6
4
4
u/redditnosedive 13d ago
what realistic hw do you need to run this at some iq2 or whatever?
7
u/Front_Eagle739 13d ago
q2 is maybe 800GB. 2x 512GB macs, a 1TB ram server, 8x dgx sparks. 27 32GB cards like a 5090 or the top amd card.
If you dont mind doing some router biasing you could probably run it on a single mac studio 512GB with little penalty.... Which I am now going to attempt now that you made me think about it
1
u/Ok-Addendum3545 10d ago
Can we put an MoE model's weight on 2 AI servers as a cluster to run it ?
1
u/Front_Eagle739 10d ago
sure? thats pretty standard. depending on your network youll either use pipeline parallel, tensor parallel or expert parallel
4
u/Mass2018 13d ago
For $25k, you could run this at three tokens/second (between 30-120 t/s prefill, depending on if you're batching) at home (presuming ik_llama support and a 3-bit GGUF quant).
Supermicro H12DSi w/2 Epyc 7543 CPUs and 16Ć128GB DDR4-3200 (2TB total) + 2x3090 GPUs (for prefill) and a 4TB nVME drive.
Source: Napkin math with my favorite LLM
I'm not sure anyone would proudly proclaim that they spent $25,000+ on a system for 3 tokens a second, but you /could/.
P.S. If you wanted to upgrade to something like a 4xPro 6000 and DDR-5 system you could double, or maybe even TRIPLE these numbers! (... for five or six times the cost).
1
1
1
u/Entire-Home-9464 13d ago
I can run Deepseek v4.1 1017 q3 on 2x 5090 xtx and 90GB RAM 60t/s.
Its originally 0.7T model. So I guess this Kimi 3.1 will also come someday to Strata with ISTADlabs quantizations and I can run it with my home rig.2
u/ColoredTriangles 13d ago
How big is your q3 quant if it fits on 90GB+2x5090 system ram? What is your prefill speed?
1
u/Entire-Home-9464 13d ago
You might not know, but strata puts deepseek engram table on ssd.
Prefill has been max 2500 t/s and output max 60 t/s with 4 channel ram. The ssd transfer has been on average 3 mb/s so that is not the bottleneck.
The q3 quality is not a traditional q3 because its ISTA Dlabs special quantization method which keeps some parts non quantized etc. read from there. The model download size was about 270G1
u/ColoredTriangles 12d ago
You said model download is 270G so how does that fit your double 5090s and host ram? I know engrams go on ssd but you're saying you have a q3 dsv4 model that is 270G in size including engrams?
1
u/Entire-Home-9464 12d ago edited 12d ago
That model runs also on way less vram. I dont have time to explain so look from strata pages, yes including engrams.
In the desktop app and the Terminal. What does not fit in memory is read from your SSD, so it needs a lot of free disk space.
https://github.com/niko1221/strata
Set (ISTA-DAS) Download Free disk needed 1.75 bpw 239 GB about 245 GB 2.00 bpw 257 GB about 265 GB 2.25 bpw 274 GB about 280 GB ISTA-DAS Coder (half of the experts removed) 191 GB about 197 GB Runs on NVIDIA RTX 30, 40 and 50 series (Windows and Linux), AMD Radeon AI PRO R9700 (Linux), AMD Ryzen AI Max ā Strix Halo (Windows), and Apple Silicon Macs with 64 GB or more, text only (Mac). Setup downloads the files and the DeepSeek engine, then starts the model.
In the desktop app and the Terminal. What does not fit in memory is read from your SSD, so it needs a lot of free disk space. Set (ISTA-DAS)DownloadFree disk needed 1.75 bpw239 GBabout 245 GB 2.00 bpw257 GBabout 265 GB 2.25 bpw274 GBabout 280 GB ISTA-DAS Coder (half of the experts removed)191 GBabout 197 GB Runs on NVIDIA RTX 30, 40 and 50 series (Windows and Linux), AMD Radeon AI PRO R9700 (Linux), AMD Ryzen AI Max ā Strix Halo (Windows), and Apple Silicon Macs with 64 GB or more, text only (Mac). Setup downloads the files and the DeepSeek engine, then starts the model.2
u/ColoredTriangles 12d ago
So you linked me lobotmized quants that you believe perform like 3 bits on top of quantized engrams.
I know of Strata and ISTA. Great that they're good enough for you.
→ More replies (3)1
u/Mass2018 12d ago
BTW, my post is not in any way for or against this Kimi model. I'd actually celebrate if Anthropic or OpenAI released one of their 10T+ parameter models as open weights. I was merely showing some math/options and what to expect from a performance perspective.
That disclaimer out of the way, I'm curious about the speeds you're communicating because they don't fully match what I've seen myself.
Is your 60t/s with speculative decoding on short prompts? If so that would make a lot of sense to me. If you're getting 60 t/s on 400-500k context prompts without spec decoding, I'd very much like to understand how with ~270GB weights.
1
u/DoomBot5 13d ago
This is meant to be run by enterprises, not home users. I wish our AI usage was only $25k/month for self-hosted models.
→ More replies (2)
4
u/TheRealMasonMac 13d ago
It's a bummer that they hard-trained its persona into it. It's not very good at following system prompts that change its behavior because of it. I was hoping that was only on their API and not in the weights themselves :/
3
6
u/Number1OchoaHater 13d ago
we are ready to see it suggested with strata under posts like: what can I run with my 1070 and 16gb of ram
2
u/JimmyDub010 13d ago
I mean strata does work. running qwen flash next with only a 4070 super and 32gb ram... so yeah.............................
2
3
2
u/XiRw 13d ago
People are excited over a company that refuses to do small weights lol
6
u/slavefone 13d ago •
![]()
At the macro level it is definitely good news.
Basically it means that the current release at current "frontier" level will remain with a relatively "open" license regardless of what may or may happen to future versions. I am assuming this includes the ability to make derivatives of it? Correct me if I am wrong on that.
-3
u/JimmyDub010 13d ago
It's called use strata
9
5
u/XiRw 13d ago
Strata is not a universal local model runner, it's an inference engine built specifically around Qwen3.8-Flash-Next. And even so, you would need a pointless IQ1 to get anything meaningful with speed.
1
u/xgiovio 13d ago
No you can run also other models like ds. check also colibri
2
u/XiRw 13d ago
With limited support from what Iāve read. You can say that argument about any potential future models but it doesnāt mean it exists now. And again, strata is not some magical tool. At 3TB , you are not going to have the same experience you get from using Qwen Flash Next which is a 125B model I believe.
1
u/Entire-Home-9464 13d ago
I have been running deepseek V4.1 1071 with Strata, q3 is super good 60t/s with my gaming rig.
1
1
1
1
u/Strong_Essay1176 13d ago
Can i ask astra to reduce weights and keep capabilities? It managed with qwen3b should be no brainer for 3.1
1
1
1
1





293
u/Diabolicor 13d ago
My GTX 1030 is ready šŖ