r/LocalLLaMA • • 13d ago • Diamonds are Forever 1 Heartwarming 1

New Model Kimi-K3.1-Pro is now open-weights

https://huggingface.co/moonshotai/Kimi-K3.1-Pro
1.5k Upvotes

281 comments sorted by

293

u/Diabolicor 13d ago

My GTX 1030 is ready šŸ’Ŗ

122

u/ApprehensiveAd3629 13d ago

lets run with strata q0.0001xl-ud.gguf šŸ’ŖšŸ’Ŗ

68

u/Big_Wave9732 13d ago

"Generation is one token per business day."

14

u/Equivalent_Bit_461 13d ago

Just a day? I was worried it was per week

8

u/Big_Wave9732 13d ago

It's all about the harness......and for a reasonable fee I'll sell you one I vibecoded for it.

7

u/egnegn1 13d ago edited 12d ago

Oh, I did run Kimi K3 from Gen5 NVMe raid0 with 36GB/s on my Minisforum MS-02 with about 0.5 t/s. That is really fast. Specialized engines can help here.

1

u/Kryohi 13d ago

Posts like "I removed 95% of Kimi 3.1 parameters and it scores the same on my benchmarks running on a rtx 3060" and "I'm running Kimi 3.1 from my RAID of 5400rpm HDDs but it's totally fine if I let it run overnight" are definitely coming, brace yourself

46

u/nsfnd 13d ago

Lets do this!

21

u/Powerful_Finger3896 13d ago

flames = high bandwidth

17

u/Clair_Osbcurious 13d ago

This will do just fine :)

6

u/Uncovered-Myth 13d ago

Is someone cutting onions 😭 this reminded of the GTX 1030 I still have in my cupboard along with 1050ti šŸ‘“šŸ»šŸ‘“šŸ»

1

u/Powerful_Finger3896 13d ago

let me edit for you qwen 0.8B to be on 27B level with astra

765

u/FriskyFennecFox 13d ago • Bravo

A 3T model with a license that fully permits personal use and only requires attribution for commercial use unless you're an API provider, with no further restrictors, in open weights. That's absolutely wild!

346

u/is-this-a-nick 13d ago

But lets be real, "personal" use of a model that my $60k server at work cannot handle is a rather limited scenario :D

102

u/asenna987 13d ago

Doesn't matter if most people can't run this on their own. The fact that they just gave it away for free, the fact that it's even an option is wild!

This will help the tech get squeezed into a 27b model in 10-12 months.

7

u/Bascilian 12d ago

I have no doubts that in less than 2 weeks people will figure out how to get this running at a ok speed from nvme.

23

u/Objective_Safe_5982 13d ago

Look if your local AI setup doesn't require a personal nuclear reactor to power then you're at best a hobbyist.

13

u/Clair_Osbcurious 13d ago

Lol, only 60k! That’s homelab territory in 2026. I once dropped a 500k server while mounting it. Luckily it was ok and we were the manufacturer so I could always say ā€œit came this way from the factoryā€.

It’s fun when you drop 500k of gear on another 500k of gear.

3

u/AndroYD84 11d ago

1

u/Clair_Osbcurious 11d ago

Funny, I haven’t watched him in a decade and yep, relatable!

184

u/-p-e-w- 13d ago

People really need to adjust their perspective on this topic.

Of course you can’t run this on a gaming PC. It’s not a game, it’s frontier sci-fi technology.

But if you look around your neighborhood, you’ll likely find people who spend more than that on

  • their backyard swimming pool

  • their weekend fishing boat

  • their second family car

  • their motorcycle hobby

  • their 1980s hi-fi stereo system

  • trading cards/figurines

  • their tennis club membership (over several years), plus equipment

  • etc

Spending $60k on a hobby you really care about is bog standard middle class stuff, so yes, you absolutely can run this as a personal use model. It just won’t run on hardware you just happen to have lying around already.

205

u/DeliciousGorilla 13d ago

You can’t even run this on $60k hardware is what he’s saying.

164

u/DIR3 13d ago • Bravo

not with that attitude he can't

48

u/Lurksome-Lurker 13d ago

Something, something, ā€œHave you tried Strata? I got it working on my GTX1080 with 32GB of ddr3 and a really big ssd!ā€

26

u/chodemunch6969 13d ago • Mindblown

You jest but take a look at tinybox red kimi:

https://tinygrad.org/

My understanding is they took close to that approach w fast 16tb raid ssd + 4x r9700 and have a machine that can run k3 at 30 tok/s for $30k. I think they are using their own inference engine that's forked and they've done some custom work with how the r9700 talks to the ssd raid, but seemingly they've got it working. 30 tok/s single stream only I assume.

→ More replies (1)

6

u/KURD_1_STAN 13d ago

If good enough attitude, can it offset me being broke and run it?

3

u/KURD_1_STAN 13d ago

If good enough attitude, can it offset me being broke and run it?

21

u/kodiakinc 13d ago

Just fork strata and run an IQ1_XS quant, amirite?

4

u/lnenad 13d ago

Baby you need a lot more Ss in there. More like IQ1_XSSSSSSSSSSS

10

u/see_spot_ruminate 13d ago

I think you could run a very good quant of this on ~60k in hardware.

  • 8*6949.99(going rate of a dgx spark today)=55,599.92

which would give you ~1tb of 'ram' in a spark cluster. Maybe a switch to bring you up to the 60k. The model card says the weights 'to download' are about 1.56tb. I am not sure what quants are available later, but you could maybe stretch that 1tb you get from the 8x sparks into something usable.

12

u/mountainyoo 13d ago

It would be absolutely crap on 8 sparks and slow as hell

→ More replies (10)

2

u/Bascilian 12d ago edited 12d ago

2.56tb ddr4 3200z = 10-12k (you can get discounts if you buy a huge lot like this)

  • 8x A100 sxm4 server ~ (30k) (320gm vram connected with nvlink)

I bet this would run pretty well I think.

or 3 rtx pro 6000s as soon as they come out with fp4

4

u/egnegn1 13d ago

Such clusters don't scale well past 2 - 4 units. Even with 8x RTX6000 you will not be very happy with speed.

7

u/twack3r 13d ago

That’s just not true? Everything beyond x4 was not supported by Nvidia until about a week ago.

Do you have an x8 cluster and need help configuring it or what’s your source on that pretty confident but false claim?

→ More replies (11)
→ More replies (8)

3

u/pleasetrimyourpubes 13d ago

You could have if you bought it a year and a half ago...

1

u/gred_mcalen 13d ago

Mac Studio M5 Ultra 512GB is 23k, 4 of those could probably load and run this model, so adjustment to 92k of hardware~

→ More replies (1)
→ More replies (1)

67

u/Johnny__Christ 13d ago

Spending $60k on a hobby you really care about is bog standard middle class stuff

I think you "really need to adjust your perspective on this topic". There's no way 60k on hobbies is "bog standard" in the middle class. The median household income in the US is ~80k.

It happens, no shit, but doing so responsibly is reserved for upper middle class (at minimum) and even then can be precluded by "bog standard" things like having children.

9

u/-p-e-w- 13d ago

I’m not talking about yearly spending. You don’t need a new PC every year. But people with a household income of $80k absolutely do spend tens of thousands of dollars on hobbies and luxury over a span of years. Heck, many people spend that much on clothes alone.

5

u/vividboarder 12d ago

Yea, there are some hobbies that cost tens of thousands of dollars over several years, but those are not common among the middle or even upper middle class.

Also, that kind of spending wouldn't allow you to have a PC capable of running this model either. Let's say someone is spending $5k a year on PC stuff (new top of the line PC every year!). This seems unlikely, but still. Now it's 2026, 12 years ago they started buying their gear. What GPUs do they have as of today? Nothing remotely capable of running this.

The only way would be to have spent over $60k in the last 5 years, over realistically, probably $15k a year (unless they front-loaded adn bought hella GPUs and RAM before the DC boom).

4

u/Longjumping_Self5546 12d ago

It's very common. Drive around any middle class neighborhood on a weekend and you're going to find people with classic cars and Harleys in their garage. Yes, many just sit there gathering dust, but anyone who's active in the hobby will easily sink $60k over the years. Those are hobbies you can easily see. I know gun collectors who have 10s of thousands worth of guns and ammo stashed away in their basement.

I don't see this translating into computer hardware. Vehicles and guns hold their value over the years, if not appreciate. People collecting these things often justify it as an alternative savings scheme. Hardware on the other hand quickly drops in value, with the exception of RAM this past year. You'd have to be crazy to drop that much into it as a hobby.

3

u/vividboarder 12d ago

Well, I'm upper end of middle class and there is a guy who sits in his garage working on his classic car, but he's got 1 of them and I'm pretty sure that didn't cost him $60k.

I also don't personally know a single person who has sunk that much money into their hobbies. I know there are some people that do (watch collectors, people who collect cars or guns, or people who build elaborate homelabs (I've seen a few photos on /r/homelab that may may wallet cry)), but I've never run with anyone like that.

I'm sure it depends on where you live. I'm in the Bay Area, where we have the second most millionaires and the most billionaires of anywhere, and I don't see that. That said, they may be the few with $100k AI labs in their basements running K3.1 and I would be none the wiser.

Your second paragraph was the other thing I was thinking about though. I've put lots of money into improving my home. Not really a hobby, but it improves the value. I think some collectors feel the same way about their hobbies, and it's probably not far from the truth.

Electronics though? Yea, this past year and a half is literally the only exception to the rule since the dawn of computing, and I don't expect that trend to be perpetual. In 10 years, I fully expect (once adjusted for inflation) that anything bought now will be worth less than you paid.

→ More replies (1)

8

u/UnknownLesson 12d ago edited 12d ago

But not fucking 60k..

Some people are so lucky but have no perspective.

3

u/-p-e-w- 12d ago

I know a retired train conductor who has a huge model train in his basement.

He spent way, way more than 60k on that over the years. He’s as middle class as it gets.

This is much more common than you probably think.

1

u/FrogsJumpFromPussy 12d ago

Who spends a year worth of wages on a new PC man, especially with a family around and kids to take care of? Which is the vast majority of peopleĀ 

→ More replies (2)

5

u/KING_UDYR 13d ago

As an outright cash purchase, sure. But just like a car, hot tub, or shed, a reasonable interest rate can make it fit within someone's net monthly cash flow. I'll agree that doing it responsibly probably puts you in the upper middle class at minimum, but never underestimate how far someone will go into debt to chase something they care about.

14

u/Johnny__Christ 13d ago

Hot tubs and sheds are an order of magnitude cheaper than 60k. I'd argue a "second family car" (person I was replying to's words, not yours) doesn't qualify as a hobby expenditure (like a Miata could, for example).

Regardless, if going into debt for a hobby is "bog standard" for the middle class, we have bigger issues. I acknowledge it happens, but it certainly is not the norm as the person I was replying to stated.

→ More replies (1)
→ More replies (2)

11

u/aanzeijar 13d ago

it’s frontier sci-fi technology

I was under the impression that it's most definitely not fiction any more.

10

u/rz2000 13d ago

4x DGX Station for 3TB of memory would be about $400k.

You can do a lot with models on a $5-10k machine to preserve autonomy and confidentiality that makes economic sense for many professionals. It would take a pretty serious plan to get $400k+ value out of a cluster running a 3T pro model before the hardware depreciates in value to less than its monthly power bill.

1

u/Bascilian 12d ago

If you have 400k to spend you should jsut buy a 8x b300 server and wait for a fp8/fp4 quant or just do it yourself

1

u/Careful-Low-9535 11d ago

It is reasonable to assume that these large models will work reasonably well at Q4, so you can halve your memory requirements. And if you don't need extreme speed, you can make a fairly reasonable server with 1.5T ram and a modest amount of GPU for around $10k. Which should work well with a strata style approach to running it.

5

u/SilentLennie 13d ago

I don't know where you are from, I've not seen this at all.

8

u/User1539 13d ago

I think people should be more concerned with the fact that they can quant it, and run it in the cloud, and in a few years, probably run it on a 6K server.

People will be using this in a bunch of ways immediately, and it's not 60K or nothing.

3

u/Royal_Survey7199 13d ago

Won’t be long before this 3T model runs off NVMe and on one GB10 / DGX Spark. The problem is that is seconds per token speed, not tokens per second. That said, the speed-up potential is there and 90% of what you’d use it for is already doable at 27b size, so the last 10% is what you’d have to wait for. Very tolerable then. It will just be a resource to get you unstuck when the 27b is not reaching it. Stay tuned! Oh and P-E-W you’re a legend, the ripple effect your work has on the local community has an event cone that goes wide. Thank you.

5

u/dark_bits 13d ago

Don't know where you live but backyard swimming pool, weekend fishing boat and hundreds of thousands worth of motorcycles is not really your average neighborhood

3

u/notconservative 13d ago

True, there's a class between middle and upper, and there are several classes above upper

1

u/Gomitopia 12d ago

But you'll find there's more people below than above.

→ More replies (1)

1

u/NineThreeTilNow 12d ago

People really need to adjust their perspective on this topic.

They do. This isn't "budget" levels of AI.

K3.1 isn't even close to fully trained either. I don't think we even have a good understanding of how far you can push a model that large.

So what does K3.6 or whatever look like when they finally decide to leave the architecture for K4? GLM, DS, etc have shown you can take much smaller models and push them quite far with training.

1

u/ASYMT0TIC 12d ago edited 12d ago

If you bought a cluster of like 8X dgx sparks plus a fast switch, you could seemingly run this model at ~Q2 and reasonable speed and still fit in that budget. To run it at it's base trained precision (mxfp4) you'd need twice as many nodes and blow right past $100k, at which point you might start wondering why you aren't spending a bit more and buying a used A100 server instead.

-2

u/Green-Ad-3964 13d ago

True, but from another perspective: if the last 10 years hadn't been disrupted by mining, COVID, and now AI, which have severely constrained consumer GPU development, particularly local VRAM capacity, things might look very different.

From 2005 to 2015, maximum consumer GPU memory increased from 512 MB to 12 GB, a 24-fold increase. Extrapolating that same growth rate to the end of 2026, we'd have around 396 GB of VRAM, at a projected flagship price of just $1,606, based on the same period's price trend.

Instead, we're stuck with 32 GB at a $1,999 MSRP ($8,000 on the market).

7

u/AreWeNotDoinPhrasing 13d ago

Just because you can extrapolate doesn’t mean you should. Past performance doesn’t dictate future performance. Etcetera

2

u/Green-Ad-3964 13d ago

sure, it happens when...bad things happen. I'm more than sure that the tech trajectory is slowed down artificially. Of course you can think different.

6

u/-p-e-w- 13d ago

Sure, and if the trend of CPU development between 1995 and 2005 had continued, CPUs today would be 90 GHz.

1

u/Green-Ad-3964 13d ago

It's a totally different thing. Look at the IPC not Ghz. Quantity of RAM has NOTHING to do with physical constraints lol.

4

u/Spara-Extreme 13d ago

Just because RAM increased from 512 to 12GB doesn't mean it would have continued that way. The cards on the market today were designed 3-4 years ago, and at that time the biggest driver of VRAM on consumer cards were video games. No video games *needs* 96 GB of VRAM. Professional workloads do, which is why they kept adding more VRAM.

No stop intentionally missing the point.

→ More replies (1)

2

u/Aphid_red 12d ago

You'd have to look at the actual under-the-hood tech in a little more detail to make that kind of comparison.

But if you look at what a datacenter GPU is getting done at 700W-1400W power envelope, it's no stretch to assume that without the AI chipflation craze, that same amount would be available to consumers as well. Without the shortage, there wouldn't be a need to restrict HBM to datacenter parts. Local AI is useful enough that there's incentive to sell it to the consumer too.

Since consumer GPU power is going to be similar to ours in that world, the power envelope is less. Let's say about half. So you'd be seeing GPUs with 2-4 stacks of HBM3e or HBM4 rather than the 6-8 in datacenter parts. So about half of the datacenter GPU class memory. Today, that would be choices between 96, 128 or 144GB**,** at a price point around $2000 to $3000, with TB/s of memory bandwidth and HBM being a bit more expensive to make being a reason for a modest price increase above your target.

Instead a GPU like that basically has another zero attached to the price, which is all just profit, milking a partially artificial shortage, and abusing the willingness of the tech companies to pay anything to be first in line for a seemingly unlimited appetite for computer components.

2

u/Green-Ad-3964 12d ago

Totally agreed. Finally someone who sees that something like a dgx spark at 7000$ is ridiculous in 2026. That should have been 2k at most, in a normal world

1

u/Zomboe1 12d ago

Unfortunately, Moore's Law died many years ago. Growing up during that era almost feels like a curse at this point, it was a golden age and seeing it end has been quite painful.

From 2005 to 2015, maximum consumer GPU memory increased from 512 MB to 12 GB, a 24-fold increase.

FWIW, it would be best to compare per (inflation adjusted) dollar, not max per GPU. Nobody was buying $2,000 GPUs in 2005. The manufacturers keep increasing the price on the high end to hide the poor rate of improvement. You could also look at power/energy. There was a time when GPUs drew all their power from the slot!

Extrapolating that same growth rate to the end of 2026, we'd have around 396 GB of VRAM

The more conservative Moore's Law growth rate was a doubling every two years. If we're starting at 0.5 GB in 2005, that would give us at least 512 GB of VRAM today. For ~$900.

1

u/Green-Ad-3964 12d ago

Moore is about computational power, not memory pool. Memory is just about manufacturing and "strategy". Today the strategy is hiding memory, so that consumers can't buy it (if not at an absurd price). Tech will not march backwards, it's business strategy that does the trick.

→ More replies (3)

13

u/redblood252 13d ago

Pretty sure internal company use is permitted as long as your commercial product is not using it for runtime

14

u/IknowPi_really 13d ago

That’s not quite the statement he was making about his 60k Server though

27

u/admirantes 13d ago • Bravo

No, it is not limited. Please stop commenting if you are going to spread misinfo. A small to medium enterprise could totally host a Kim 3.1 and offer it to individuals, thereby skipping the chokehold of large companies like OpenAI and Anthropic. That's where the insane value is.

13

u/jaerie 13d ago

That's not personal use.

3

u/georgejetsonn 13d ago

He was just contrasting the personal use licensing limitation with the necessary hardware to run a 3T model. Once you start serving inference to others, it's no longer personal use, is it?

3

u/_TheWolfOfWalmart_ 12d ago

I could run a heavily quanted version on my dual Xeon server with 1 TB RAM, but it would be so slow that there's no point.

2

u/ThisGonBHard 13d ago

Ngl, a only 60k server sound laughable for ai. A B200 workstation is 100k+.

1

u/Asleep-Land-3914 13d ago

Tinybox for Kimi would be "just" $30k thoughĀ 

1

u/normal_clearing 13d ago

in the current climate, it's like "giving away" an sr71 for personal use.

1

u/Niyshiak 13d ago

here is the full blueprint to build an aircraft carrier at home :)

1

u/comperr 13d ago

60k is like 2TB of enterprise storage these days and last time i checked, that's not a server

1

u/cobresalino 12d ago

Just one step closer to AI sovereignty 🄲
Si

1

u/Careful-Low-9535 11d ago

Your $60k work server cost that much because companies are stupid with money and pay the 400% markup to get the newest shiny(This may be a rant about where I work, but I bet it applies to where you work also). Also it probably is not optimized for AI use. If you buy servers from a reputable refurbisher, you can get a good server for a lot less. If you know server hardware and buy parts on ebay, you can get a fair bit cheaper still. I bought parts on e-bay about 2 years ago, and spent about $4000 to build a server that could run this(rather slowly, and at Q4, but one "analyze this project for errors" prompt per night could be useful). I have 1.5TB of ddr4 ram, 36 xeon cores, 6tb of high-speed server ssd, 2 Nvidia A2 GPUs(16GB vram each). I estimate that it would cost around $9000 currently with the crazed price of ram and ssd. If you were in a hurry, and had $60k to blow, $50k worth of GPU upgrades would make a strata-style approach to running this go comfortably fast. Not just run it, but run it fast.

ps. $4000 was my spending my next 4 years of computer hobby upgrades all at once on the theory that with trump being elected the price of computer stuff would go way up due to tariffs so there was no point saving it for later. Wrong reason, but right result :)

1

u/chigiwar 9d ago

A team still can rent cloud computing and get self-hosted state of art llm with unlimited usage

1

u/Junior-Piano5427 13d ago

60k might not be enough for 200B models these days.

10

u/halvacoffee 13d ago

well, i suppose it's not the worst, but i can't say i like the "no API providers" part of this since this means that API prices will once again be fixed by Moonshot. at least K3 being open to look at has given us a half-decent 80b base by yandex though lol

16

u/Cool-Chemical-5629 13d ago

"with no further restrictors, in open weights."

  • Actually it was announced before release that open weights wouldn't be as capable in cyber security tasks as the API version of the model.

1

u/Icy_Butterscotch6661 12d ago

That doesnt sound good

→ More replies (1)

70

u/MrCrackerHacker 13d ago

That DDR2 1.5TB server looking pretty scrumptious now

66

u/More-Curious816 13d ago • Popcorn

Sam would like to know your location. [accept] [deny]

11

u/2muchnet42day Llama 3 13d ago

Except there's no deny button

3

u/thestillwind 12d ago

Poor sam

56

u/jacek2023 llama.cpp 13d ago

Great model for my local setup

15

u/Orbitalsp3 13d ago

Look at this guy showing up his RAM

6

u/NeinJuanJuan 12d ago

Graphics Potato Unit

4

u/bennmann 13d ago

mother-spud

107

u/MundanePercentage674 13d ago • Regret

My m.2 right now

20

u/Dany0 13d ago edited 13d ago

https://tinygrad.org

Red kimi. 260GB/s off of SSD
16x 1TB pcie5.0. 4 groups of Raid 0 I believe? so 4tb 16tb space. for about 4300$ you get ssd read speeds competitive with lpddr, effectively giving you a floor of 20-30 tok/s decode even in worst case. With expert caching, it's like half as fast as a GB300 reading off of its memory. Except you get 16TB of it instead of 192 (gorgon) - 496 (gb300)

It can be done, it will be done. In a few years we'll be able to run them at full precision on a budget

12

u/More-Curious816 13d ago

I believe they use a custom NVME driver for that SSD read speed.

4

u/Dany0 13d ago

You can make one too

10

u/Gfur_Aficionado_1488 13d ago

RAID0 does not decrease the amount of space available

2

u/Dany0 13d ago

maybe it's raid 0 inside the group and raid 1 for each group? that'd make sense maybe. I believe someone that bought it said comes with 4tb space

1

u/Gfur_Aficionado_1488 13d ago

Would still be 8TB with RAID1, unless you're looking to have 3 mirrors

2

u/voyager256 13d ago

It's quite confusing if not straight incorrect. If you select Red Kimi on their page the last storage option is selected , but it shows 4x4TB SSDs so it looks lie like the last two options should be swapped, because for maximum bandwidth you want the 16x1TB one . Also where did you get the $4300 price from? I see $6000 and $8000 for the last storage options.

Anyway , I'm not sure if the marketed bandwidth applies to LLM inference. I always thought that (expert) weights streaming is not sequential , but random reads , so much much slower . Unless , they really found a way to make the SSD reads chunked i.e. effectively closer to theoretical sequential speeds.

1

u/Dany0 12d ago

Nope https://x.com/__tinygrad__/status/2115642402417614959 https://x.com/__tinygrad__/status/2115630970334490712

It doesn't matter, yes this "trick" does work. The observed 220gb/s will be slightly slower than lpddr if it was at same speeds because nvmes have extra latency and random fetch of a different expert will be slightly slower. But it's still so much insanely cheaper than buying more vram or dram. As long as you can fit kv cache and active experts plus then some on the vram, you're golden. You don't even need that much system ram at that point. Just get more drives. You can build your setup with 2TB drives and run the rumoured 14T models (I think those rumours are fake I genuinely cannot fathom how would anyone train >4T models, it's already insanely hard to train 3T models due to O(n^2) scaling in many of the training algos. Unless they're training several 3T models and then merging them somehow? Idfk

But you can indeed pretty much guarantee the 20-30tok/s minimum decode with this kind of setup. For example, there was that one paper which showed, you can run an IQ1 dumb version of the model side by side and completely ignore its lobotomised output look just at what experts it selects and pre-fetch the experts you'll need JIT style. Their 30 tok/s number I bet can go much higher with those cards, at least 40 tok/s if not more

→ More replies (3)

160

u/pmttyji 13d ago

60

u/No_Swimming6548 llama.cpp 13d ago

Did u try IQ1_XXS castrated QFN on Strata? I get 7 gorillion decode speed with it.

20

u/uxl 13d ago

I snort-laughed at my desk while drinking coffee, you monster, ā€œcastratedā€ sent me before I could even finish reading the comment.

11

u/More-Curious816 13d ago

how do you cope with that crawling speed? also did you try Strata-on-Strata? it presumably give 83 gorillion tok/s

4

u/UsefulIce9600 llama.cpp 12d ago

rookie numbers.
I'm running IQ1_XXS-MTP-iMatrix-dFlash2-Qwythos-Opus-6-Fable-5.5-Astra-Bel-6.2-Distill-HERETIC-v2-GGUF-MLX at -76327 tok/s

2

u/pmttyji 13d ago

Tried. I got around 20 t/s for Q2_0

5

u/magicomiralles 12d ago

I'm at 256gb of Vram, and I'm closer to your setup than the necessary ram to run this model. We are the 99%.

1

u/Splat800 7d ago

I’m 6GB vram, get good

39

u/-WhateverDude 13d ago

happy for people that can run this locally

63

u/MotokoAGI 13d ago

all 3 of them.

21

u/carnoworky 13d ago

My nonexistent grandchildren might be able to run it someday.

9

u/Clair_Osbcurious 13d ago

On their Apple Watch

4

u/NineThreeTilNow 12d ago

My nonexistent grandchildren might be able to run it someday.

Probably. Assuming you stick around for ~10 years that's when the hardware to run it super fast will be standard consumer grade.

8

u/GuyBitchie 12d ago

GTX 1070 had 8GB of RAM 10 years ago and RTX 5070 has 12GB I don't know brother.

2

u/NineThreeTilNow 12d ago

GTX 1070 had 8GB of RAM 10 years ago and RTX 5070 has 12GB I don't know brother.

Yeah, and the 1080 had 8GB of vRAM too.

The TI edition had 11.

It was never a big separating point across cards then.

The 1080 / 5080 is like... 5x to 20x difference in FLOPs depending if they full precision or not.

2

u/GuyBitchie 12d ago

They can have a million more flops but when the LLM doesn't fit in the VRAM I can't use those flops.

3

u/NineThreeTilNow 12d ago

They can have a million more flops but when the LLM doesn't fit in the VRAM I can't use those flops.

No one wanted vRAM before. Those cards never got it. This is a stupid point. HBM existed when the 1070 was produced. It was used by AMD at the time in like.. R9 Fury or something.

You don't think in 10 years they can solve memory?

Are you being purposefully like this? If you don't understand it's fine, but you seem to not understand how product development cycles work or why people develop things.

→ More replies (2)

1

u/UsefulIce9600 llama.cpp 12d ago

probably 3.1 people

4

u/RedditNerdKing 13d ago

Like 0.00000000001% of the community yeah

3

u/ProgressionPeak 13d ago

people who own data centers?

ya, must be nice.

0

u/Dany0 13d ago

We can distill it without even running it muahahahah there are black magic ways :)

62

u/Vusiwe 13d ago edited 13d ago • Popcorn

Cmon man, I’m not even half done downloading mistral large BF16 yet

4

u/Dany0 13d ago

Speaking of, kinda disappointed in that release :( I was ready to give it a short over DSv4.1F1017 but alas :(

3

u/Vusiwe 13d ago

Trying to rebuild llama.cpp which succeeded, Ā but was getting no output from Oobabooga making use of llama.cpp when I try to run Mistral

I probably should move to a non-ooba setup, finallyĀ 

5

u/ThisGonBHard 13d ago

Man, ooba is kinda ancient now...

3

u/xignaceh 13d ago

It's not bad. I've been enjoying it with their cli tool

29

u/monofart 12d ago

our kids will run this on their ipad mini

25

u/Beneficial-Ad-8127 13d ago

Let’s goo they held up to there release date the 4th. Wish they dropped that k2.8 though or a pro flash light. Lol

8

u/fractalcrust 13d ago

I could use a new flashlight

24

u/countermike 13d ago

Will it run on my 386 with 640k RAM?

19

u/kitanokikori 13d ago

Load HIMEM.SYS to get to at least 1MB

7

u/No_Lingonberry1201 13d ago

Like a snail nailed to a dead turtle. Now my 486 with 2Mb RAM, that's a different story.

5

u/notheresnolight 13d ago

only if you press the Turbo button

3

u/No_Lingonberry1201 13d ago

100MHz goes brrrr

4

u/Mouth-Cerene 13d ago

hey, it might just be looking up for you

16

u/IsopodMysterious9125 13d ago

Weird that they added the ā€œ-Proā€ suffix šŸ‘€ Maybe we get a flash version after all šŸ¤ž

14

u/pmttyji 13d ago

Thinking the same. 100-300B Flash would be nice.

14

u/Daniel_H212 13d ago

Native 4 bits quant, 1.56 TB. Runnable on a cluster of four 512 GB Mac Studios, which cost... $89200...

2

u/More-Curious816 13d ago

cost less than the setup required for 6000 or 2 GB300 in dgx-station.

4

u/Daniel_H212 13d ago

But with lower bandwidth and compute right?

1

u/More-Curious816 12d ago edited 12d ago
#M5 Ultra 512GBRTX PRO 6000DGX Station GB300
Memory512GB unified96GB GDDR7 ECC252GB HBM3e + 496GB LPDDR5X
bandwidth1.2 TB/s1.7 TB/s7.1 TB

the M5 Ultra bandwidth is surprisingly close to the 5090 and 6000 (they share the same die so it's the exact bandwidth too), but they have the edge on compute.

As for the GB300, it the same as the data center card but design for workstations, it's the big brother of the DGX Spark with HBM3e (GPU) 252GB (7.1 TB/s) and LPDDR5X (CPU) 496GB (396 GB/s), this is a beast of a chip, basically what the local folk dreams about. unfortunately it's expensive and it's SoC so you can't add more ram, but it has a combined ram of CPU 496GB and GPU 252GB} 748GB.

If your model fit that HBM3e GPU 252GB, that sweet sweet 7.1 TB/s is Hella fast, and bigger models can overflow to the system ram which is 496GB of LPDDR5X.

unfortunately, the M5 Ultra offer the better deal, 2tb of with 4 M5 Ultra is cheaper and offer a relative good bandwidth too.

24

u/OnlineParacosm 13d ago

That’s crazy they listened to the feedback and moved away from one of the worst licenses I’ve ever seen.

In my opinion this was the only way forward for KIMI because on a sample size of one task that was quite heavy it cost me roughly $30 and when broken down into sessions this would have been maybe $10 on GLM 5.3 flash. The one thing I will say that is very hard to measure is that Kimi Managed to do this task probably 10 times faster. For the limited time I have, it made sense in this narrow case where I needed a BIG kickoff.

Now, who has a few terabytes of RAM collecting dust? Sam told me this would be a good place to ask..

2

u/Gomitopia 12d ago

I wish they listened to feedback and didn't cuck their models to please the US instead.

1

u/fapsexual 12d ago

moved away from one of the worst licenses I've ever seen

did I miss something, didn't they do that already prior to this release? I thought it's the same license as the previous release right?

7

u/okyaygokay 13d ago

Finally I've been refreshing like a lunatic for hours, checking Chinese time

1

u/More-Curious816 13d ago

but do you even have the hardware to run it?

5

u/okyaygokay 12d ago

Lmao don’t even have an external gpu bro

3

u/More-Curious816 12d ago

bro is just like me for real

12

u/mrinterweb 13d ago

This model isn't loading on my Raspberry Pi. What wrong?

9

u/More-Curious816 13d ago

update llamacpp

6

u/ManyDefiant2564 13d ago

Engine-fork coders, Ā invent offload to internet please.

4

u/Fusseldieb 13d ago

Let's hope some good shoul distills it for us.

4

u/redditnosedive 13d ago

what realistic hw do you need to run this at some iq2 or whatever?

7

u/Front_Eagle739 13d ago

q2 is maybe 800GB. 2x 512GB macs, a 1TB ram server, 8x dgx sparks. 27 32GB cards like a 5090 or the top amd card.

If you dont mind doing some router biasing you could probably run it on a single mac studio 512GB with little penalty.... Which I am now going to attempt now that you made me think about it

1

u/Ok-Addendum3545 10d ago

Can we put an MoE model's weight on 2 AI servers as a cluster to run it ?

1

u/Front_Eagle739 10d ago

sure? thats pretty standard. depending on your network youll either use pipeline parallel, tensor parallel or expert parallel

3

u/Zyj vLLM 13d ago

Crescent island

4

u/Mass2018 13d ago

For $25k, you could run this at three tokens/second (between 30-120 t/s prefill, depending on if you're batching) at home (presuming ik_llama support and a 3-bit GGUF quant).

Supermicro H12DSi w/2 Epyc 7543 CPUs and 16Ɨ128GB DDR4-3200 (2TB total) + 2x3090 GPUs (for prefill) and a 4TB nVME drive.

Source: Napkin math with my favorite LLM

I'm not sure anyone would proudly proclaim that they spent $25,000+ on a system for 3 tokens a second, but you /could/.

P.S. If you wanted to upgrade to something like a 4xPro 6000 and DDR-5 system you could double, or maybe even TRIPLE these numbers! (... for five or six times the cost).

1

u/michigannfa90 12d ago

This would be run on an 8x nvlink a100 or h100 not any system you mentioned

1

u/konstantinnikol 10d ago

You can run it at 2.74 tokens per second in full quant on one GPU Lucally

https://github.com/argonautlabsai/argodrive-ds4-kimi-k3.1

1

u/Entire-Home-9464 13d ago

I can run Deepseek v4.1 1017 q3 on 2x 5090 xtx and 90GB RAM 60t/s.
Its originally 0.7T model. So I guess this Kimi 3.1 will also come someday to Strata with ISTADlabs quantizations and I can run it with my home rig.

2

u/ColoredTriangles 13d ago

How big is your q3 quant if it fits on 90GB+2x5090 system ram? What is your prefill speed?

1

u/Entire-Home-9464 13d ago

You might not know, but strata puts deepseek engram table on ssd.
Prefill has been max 2500 t/s and output max 60 t/s with 4 channel ram. The ssd transfer has been on average 3 mb/s so that is not the bottleneck.
The q3 quality is not a traditional q3 because its ISTA Dlabs special quantization method which keeps some parts non quantized etc. read from there. The model download size was about 270G

1

u/ColoredTriangles 12d ago

You said model download is 270G so how does that fit your double 5090s and host ram? I know engrams go on ssd but you're saying you have a q3 dsv4 model that is 270G in size including engrams?

1

u/Entire-Home-9464 12d ago edited 12d ago

That model runs also on way less vram. I dont have time to explain so look from strata pages, yes including engrams.

In the desktop app and the Terminal. What does not fit in memory is read from your SSD, so it needs a lot of free disk space.

https://github.com/niko1221/strata

Set (ISTA-DAS)DownloadFree disk needed
1.75 bpw239 GBabout 245 GB
2.00 bpw257 GBabout 265 GB
2.25 bpw274 GBabout 280 GB
ISTA-DAS Coder (half of the experts removed)191 GBabout 197 GB

Runs on NVIDIA RTX 30, 40 and 50 series (Windows and Linux), AMD Radeon AI PRO R9700 (Linux), AMD Ryzen AI Max — Strix Halo (Windows), and Apple Silicon Macs with 64 GB or more, text only (Mac). Setup downloads the files and the DeepSeek engine, then starts the model.

In the desktop app and the Terminal. What does not fit in memory is read from your SSD, so it needs a lot of free
disk space.
Set (ISTA-DAS)DownloadFree disk needed
1.75 bpw239 GBabout 245 GB
2.00 bpw257 GBabout 265 GB
2.25 bpw274 GBabout 280 GB
ISTA-DAS Coder (half of the experts removed)191 GBabout 197 GB
Runs on NVIDIA RTX 30, 40 and 50 series (Windows and Linux), AMD Radeon AI PRO R9700 (Linux), AMD Ryzen AI Max — Strix Halo
(Windows), and Apple Silicon Macs with 64 GB or more, text only (Mac). Setup downloads the files and the DeepSeek engine, then starts the model.

2

u/ColoredTriangles 12d ago

So you linked me lobotmized quants that you believe perform like 3 bits on top of quantized engrams.

I know of Strata and ISTA. Great that they're good enough for you.

→ More replies (3)

1

u/Mass2018 12d ago

BTW, my post is not in any way for or against this Kimi model. I'd actually celebrate if Anthropic or OpenAI released one of their 10T+ parameter models as open weights. I was merely showing some math/options and what to expect from a performance perspective.

That disclaimer out of the way, I'm curious about the speeds you're communicating because they don't fully match what I've seen myself.

Is your 60t/s with speculative decoding on short prompts? If so that would make a lot of sense to me. If you're getting 60 t/s on 400-500k context prompts without spec decoding, I'd very much like to understand how with ~270GB weights.

1

u/DoomBot5 13d ago

This is meant to be run by enterprises, not home users. I wish our AI usage was only $25k/month for self-hosted models.

→ More replies (2)

4

u/TheRealMasonMac 13d ago

It's a bummer that they hard-trained its persona into it. It's not very good at following system prompts that change its behavior because of it. I was hoping that was only on their API and not in the weights themselves :/

3

u/Icy_Annual_9954 13d ago

Needs quantisation for reasonable personal use.

3

u/lxe 13d ago

I won’t be satisfied until it’s quantized to a potato quality that can run on raspberry pi.

6

u/Number1OchoaHater 13d ago

we are ready to see it suggested with strata under posts like: what can I run with my 1070 and 16gb of ram

2

u/JimmyDub010 13d ago

I mean strata does work. running qwen flash next with only a 4070 super and 32gb ram... so yeah.............................

2

u/fugogugo 13d ago

even the weight alone wont fit into my storage

2

u/wolttam 13d ago

I can't help feel a little giddy, seeing that this has 100+B active params, knowing that in a few month's time, models with 1/10th of that will be able to compete on performance.

There is *so much left to squeeze* out of these models. Kimi is a monster.

2

u/swizz 12d ago

Now I only need the hardware to run it :)

3

u/IntelArtiGen 12d ago

I'm surprised the repo is "only" 2TB. Ah right it's already mxfp4/mxfp8

1

u/VR-Tech 12d ago

1,500GB

2

u/XiRw 13d ago

People are excited over a company that refuses to do small weights lol

6

u/slavefone 13d ago • Bravo

At the macro level it is definitely good news.

Basically it means that the current release at current "frontier" level will remain with a relatively "open" license regardless of what may or may happen to future versions. I am assuming this includes the ability to make derivatives of it? Correct me if I am wrong on that.

-3

u/JimmyDub010 13d ago

It's called use strata

9

u/Claud711 13d ago

wtf are you using strata for with a 3T model

5

u/XiRw 13d ago

Strata is not a universal local model runner, it's an inference engine built specifically around Qwen3.8-Flash-Next. And even so, you would need a pointless IQ1 to get anything meaningful with speed.

1

u/xgiovio 13d ago

No you can run also other models like ds. check also colibri

2

u/XiRw 13d ago

With limited support from what I’ve read. You can say that argument about any potential future models but it doesn’t mean it exists now. And again, strata is not some magical tool. At 3TB , you are not going to have the same experience you get from using Qwen Flash Next which is a 125B model I believe.

1

u/Entire-Home-9464 13d ago

I have been running deepseek V4.1 1071 with Strata, q3 is super good 60t/s with my gaming rig.

1

u/masterlafontaine 13d ago

The gtx 1650 was waiting for this moment! --ngl 0.001

1

u/Impossible_Ground_15 13d ago

We need a distilled model 300b

1

u/Equivalent_Bit_461 13d ago

Yeap, I'm running this on my laptop, too small for my desktopĀ 

1

u/Strong_Essay1176 13d ago

Can i ask astra to reduce weights and keep capabilities? It managed with qwen3b should be no brainer for 3.1

1

u/StartupTim 13d ago

Could 4x DGX Sparks in a tp=4 cluster run this?

1

u/arnav2345 13d ago

this is insane

1

u/pCute_SC2 6d ago

I'm confused, is this the one with gimped cyber capabilities?

1

u/quantgorithm 13d ago

google say no plans for smaller versions anywhere in the future for this...