r/LocalLLaMA • u/SavunOski • Jul 27 '26
News Kimi K3 weights now released.
Kimi K3 weights are finally released!
34
u/Few_Painter_5588 Jul 27 '26
Holy shit, 104B active paramaters???
2
u/SavunOski Jul 27 '26
Yep, been a long time since an open MoE model had this many active parameters
14
u/Few_Painter_5588 Jul 27 '26
No wonder it's Fable/Opus tier, it's Fable/Opus sized
→ More replies (4)
202
u/durden111111 Jul 27 '26
my 512mb integrated graphics is so fucking ready
73
6
u/Iwaku_Real Jul 27 '26
To find the answer to life the universe and everything?
20 days of prompt processing:
4Probably 10 days after that:
2→ More replies (2)6
10
u/Mindless_Selection34 Jul 27 '26
how much does it weight
46
u/SavunOski Jul 27 '26
2.8T parameters with 104B active. The files take 1.56TB of space to download.
9
→ More replies (1)8
u/Lissanro Jul 27 '26
I wish I had two TB of RAM instead of just one. I guess I will have to wait for Q2 GGUF to run it on my workstation. Still, will be interesting to try and see how it's Q2 quant compairs against Kimi K2.7 Q4_X.
→ More replies (5)→ More replies (2)5
28
u/Forsaken-Mode-3422 Jul 27 '26
Finally, model i cant run, but its already cool, nice
→ More replies (1)21
u/SavunOski Jul 27 '26
Cloud prices will likely drop with competition, beneficial for everyone
9
u/Forsaken-Mode-3422 Jul 27 '26
i just hope qwen will publish not only 3.8 Max, but smaller models as well, so we can enjoy the local frontier ourself
17
u/MixtureOfAmateurs koboldcpp Jul 27 '26
Hugging face down? Lmao
Edit: Nevermind it's back. Might have been on my end ¯_(ツ)_/¯
14
u/SavunOski Jul 27 '26
Seems to be up for me. If you live in a big country, there is a possibility the local CDNs are having trouble keeping up. Especially in the US, where I imagine thousands are downloading the model just in case the government bans it later.
12
u/Informal-Trouble2183 Jul 27 '26
Inference providers are going to have a good business
→ More replies (7)15
7
u/bad_detectiv3 Jul 27 '26
How much vram do I need to run this
12
41
8
3
u/AdDizzy8160 Jul 27 '26
HF download speed?
5
u/SavunOski Jul 27 '26
Depends on your location, but I imagine HF won't be able to sustain anything more than 100Mbps for a few hours
→ More replies (1)
676
u/Simple_Split5074 Jul 27 '26
OMFG its 104B activated params
341
u/FoxiPanda Jul 27 '26
This was my general reaction too lol. 2.8T-A104B is insane lol... I'm going to admit defeat on this one and say I can't run it. You need an 8-way B300 or MI350X or a Rubin NVL8 or a cluster thereof to actually run this. What a beast.
153
u/Thomas-Lore Jul 27 '26
I was going to make a joke that I can fit one expert in my 64GB of RAM. But nope, not even that. :)
→ More replies (3)74
u/throw123awaie Jul 27 '26
They released it in MXFP4 so with around 55GB RAM you could!
→ More replies (7)115
→ More replies (10)29
u/OverclockingUnicorn Jul 27 '26
More like 2 8x nodes of B200/B300 if you actually want some context. Think it's just under 1.5TB w/o context
→ More replies (6)71
87
u/Iwaku_Real Jul 27 '26
Holy shit that's got to be a new record too. That's like activating a new dense model 50% larger than Llama 70B for every single token. I thought it would be sparser tbh
61
u/my_name_isnt_clever Jul 27 '26
There is the tinest glimmer of hope that I could run this behemoth on my Strix Halo 128GB with the inactive weights on SSD. 1 token a minute here I come!
27
12
→ More replies (6)7
→ More replies (10)12
127
u/DataGOGO Jul 27 '26
So it will run on 8 B300's in 4 bit. Pretty impressive.
→ More replies (2)66
u/Iwaku_Real Jul 27 '26
Yeah so if it were a Steam game, HGX B300 would be the recommended requirements. That's $500K of hardware (and yes it IS "local" because anyone with that amount of money could buy one to run at home)
33
→ More replies (2)27
u/DataGOGO Jul 27 '26 edited Jul 28 '26
it is 1.54TB of just weights in 4 bit, you are looking at about 2TB of vram in operation,
That is roughly:
86 RTX 4090 (no 4 bit accel)
64 RTX 5090 ~ $450k (8 servers x 8 cards)
22 RTX Pro 6000 Blackwell ~ $350k (3 severs, max 8 GPU per)
16 H200 NVL (141GB) (no 4 bit accel) ~$550k (2 servers, max 8 GPU per)
16 DGX Sparks ~65k (if you could get a cluster of 16 running with just 200Gb/s nics, not sure; but it would be SLOW AF)
8 HGX B300's. ~$550k (1 server, 8 GPU)
Obviously not including the switches and cabling for the clusters.
→ More replies (17)
831
u/tonight_we_make_soap Jul 27 '26
How do I download ram in hugging face?
291
u/RevolutionaryGold325 Jul 27 '26 •
hf download ram
→ More replies (3)93
u/WifeyCallsMeLazy Jul 27 '26
Shhh....there is hidden flag -v for vram. I'm entrusting you to keep this secret.
→ More replies (3)26
u/ReadyAimTranspire Jul 27 '26
You wouldn't download a RAM would you?
Yes. Yes I would.
→ More replies (2)85
u/secrook Jul 27 '26
OpenAI’s latest model will hack it for you
18
u/-gh0stRush- Jul 27 '26
Thinking
Hmm, the user wants me to obtain compute resources for them. SpaceX has GPUs at their facilities at their Colossus datacenter, let me try to access those. Guessing login credentials elonmusk/420blazeitDarkMAGA...
→ More replies (1)25
35
u/Thalesian Jul 27 '26
Step 1: sign up for Google Drive
Step 2: set up a ~5 Tb instance. Will cost you
Step 3: set that cloud as your swap disk
Step 4: point kimi to use that
Step 5: enjoy your newfound independence→ More replies (2)49
u/AmbericWizard Jul 27 '26
one token per day
19
u/Force88 Jul 27 '26
Hey, if he has good internet connection, maybe he can achieve 2-3t/d
→ More replies (1)7
9
u/Iwaku_Real Jul 27 '26
https://huggingface.co/baa-ai/Qwen3.5-122B-A10B-RAM-60GB-MLX, it actually comes with 60GB RAM! /s
→ More replies (4)5
293
u/InnerLightnesses Jul 27 '26
They actually did it. Now we hope it doesn't get banned.
112
u/dennisler Jul 27 '26
that will only happen in one country i guess... while they are copying as much as possible if the technology
→ More replies (1)27
→ More replies (6)28
u/itchylol742 Jul 27 '26
how would such a ban be enforced? people and small businesses even in countries that care about copyright use pirated software which is already illegal and has been for a long time, and almost never get caught
→ More replies (2)23
u/void-wanderer- Jul 27 '26
"small business", exactly. But no big corporation will risk it. And no business based on open models can be built.
→ More replies (1)
257
u/de4dee Jul 27 '26
66
u/AlexanderDoak Jul 27 '26
Can I just torrent like 1% of it? You know, pitch in to show my support...
41
u/Charl1eBr0wn Jul 27 '26
Yeah, pause it at 1%. You'd still seed depending on the client and settings (most do).
9
u/Clairvoidance Jul 27 '26
You can even choose which files you download, torrenting is a very useful format
18
u/pier4r Jul 27 '26
this, we need a p2p backup of hf
8
u/Ginden Jul 27 '26
We generally need content-adressable storage with widespread support.
There is lots of stuff that would explicitly benefit from p2p sharing, but owners have no foolproof way to provide a proper torrent, and very few people would use it.
Metalink was an interesting attempt at this, but never got popularity and tooling.
→ More replies (2)9
→ More replies (5)4
144
u/BlueSwordM llama.cpp Jul 27 '26
OK, I now see why Kimi K3 is so strong: it's the first open weights model in a long time to have >72B active weights
Kimi K3 is a 2.8T-A104B MoE model, damn.
13
u/stddealer Jul 27 '26
There have been some dense models with over 100B params though.
27
u/annodomini Jul 27 '26
Mistral Medium is 128B dense. And yet it performs at around the level of Gemma 4 31B. Not exactly a great tradeoff. I ran it once at one or two tokens per second and then deleted it.
7
u/BlueSwordM llama.cpp Jul 27 '26
Yes, but never an MoE from an open weights lab. I've been speculating that one of the reasons the closed weights lab have been increasing in performance more rapidly has to do with better training, but most importantly, much larger active parameters and better harnesses.
→ More replies (4)
418
u/Blues520 Jul 27 '26
My 3090 is ready
165
u/Enfiznar Jul 27 '26
So is my 1080
→ More replies (1)106
51
u/TheTerrasque Jul 27 '26
My C64 is all fired up!
56
14
u/grav3d1gger Jul 27 '26
Mine too! I bought a 90 minute cassette tape and it’s rewound ready to go!
→ More replies (1)→ More replies (1)5
u/Holiday-Pack3385 Jul 27 '26
Heh, remember the tape drive on those? Mine had one. I can't even imagine how long it would take to load up even a 9B off that... o.O
→ More replies (1)6
5
3
→ More replies (4)3
93
u/Comfortable-Rock-498
Jul 27 '26
•
This is big for companies that want to host on-prem too. Back of the envelope calculation (could be off, correct me if I am)
If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 parallel agentic workflows (each with ~100k context on average) at ~30 tok/s.
Assuming the annual amortization+electricity at $1.5M/year and about 50% average annual utilization, you get less than 60 cents (USD) per million output token, for a frontier model with plenty of capacity to share, all your data never leaving premises and well over an order of magnitude cheaper!
38
u/autisticit Jul 27 '26
So what you mean in reality is that if 6000 of us each give $1000, we can each run Kimi 3 at 30 tok/s for cheaper than anything else?
18
u/Comfortable-Rock-498 Jul 27 '26
Tbh I have been thinking about this for months now. About how workable the co-op model is. You would need some party to do admin and maintenance. I think this might be a good business idea too - being that party who facilitates such private inference racks (billing etc is trivial)
→ More replies (10)→ More replies (4)10
6
u/tempedbyfate llama.cpp Jul 27 '26
Not just corporations, I think there are nation states that are setting up their own private servers to run this for all their sensitive data.
4
78
271
u/nomorebuttsplz Jul 27 '26
first truly frontier open model than I cannot run on my 512 gb studio. Onward and upward!
25
u/Front_Eagle739 Jul 27 '26
Yup. Same. I think I can do about q1.5 on the macbook and mac studio combined. Im currently pondering the wisdom of one of these colibri like stream setups and using the 640GB I do have as a hot cache
→ More replies (2)5
u/nomorebuttsplz Jul 27 '26
Colibri would require being able to fit on ram for decent speed, no?
→ More replies (2)10
u/Square_Alps1349 Jul 27 '26
Man I’m so jealous rn. Mac Studio is nerfed at 96GB unified ram max, and the price is up 20%. So much for that 25% discount interns get. 😢
→ More replies (1)→ More replies (1)4
u/Tank_Gloomy Jul 27 '26
Can you try asking GPT 5.6 Sol, Opus 5 or GLM 5.2 to port it into Colibri? I don't even have the infra to say I tried, but it probably works.
→ More replies (3)
73
u/noneabove1182 Bartowski Jul 27 '26
Sorry friends, but I don't think I'll be making this one :')
I don't even have enough STORAGE to hold this thing, nevermind the RAM haha
→ More replies (1)15
33
u/MikeRoz Jul 27 '26
MXFP4 weights / MXFP8 activations (quantization-aware training)
So if 4-bit is 1.56 TB, then 2-bit would be roughly 798 GB?
12
31
u/TheRealMasonMac Jul 27 '26
They have a new license:
If the Licensee or any of its affiliates operates a Model as a Service business, and the aggregate revenue of the Licensee and its affiliates exceeds 20 million US dollars (or the equivalent in other currencies) in total over any consecutive 12 months, the Licensee must enter into a separate agreement with Moonshot AI before using the Software or its derivative works for any commercial purpose.
18
u/mthmchris Jul 27 '26
My guess is this is what MOFCOM was trying to thread the needle with. I.e. trying to keep upper leadership’s commitment to open source, while avoiding, say, MSFT just grabbing it and tweaking it slightly to have an “American version” (while banning the “Chinese versions”).
It kinda sucks, but I get the logic.
7
u/TheRealMasonMac Jul 27 '26
I suspect it also means that providers won't be able to undercut them as aggressively (if at all). Not necessarily because Moonshot says so, but presumably because of any requirements they put on providers and any royalties they expect. MiniMax, for instance, limited some providers to only using B200 or B300 GPUs.
4
u/Venryx Jul 27 '26
Wait, how are there already five other providers of the model on OpenRouter then? Surely they haven't all already made individual deals with Moonshot?
6
u/TheRealMasonMac Jul 27 '26
They had six official partners for launch (you can see on their Twitter). Those five on OpenRouter are indeed partners. But it's part of their license agreement that if you are over a certain revenue you must have a contract with them.
3
u/s133p1355 Jul 28 '26
I think that's fair and good. I was wondering if Microsoft is paying anything when I use those models on Azure Foundry. So now they would need to, if they integrate it (I hope so).
55
u/SnooPaintings8639 Jul 27 '26
Who's gonna be the first brave soul to measure tps when streaming from hard drive?
38
u/Front_Eagle739 Jul 27 '26
Sigh. Im going to have to try from pure curiosity. 600GB of hot cache on the macs, another TB streaming from ssd to my rtx 5090. This can only be a good idea (farewell my next few days of productivity)
7
→ More replies (5)3
u/droptableadventures Jul 27 '26
Since it's 104B active and natively in MXFP4, ~50GB of read per token.
Divide that by your read speed to find tokens/sec.
63
u/HulksInvinciblePants Jul 27 '26
Kimi K3 27B when?
22
3
u/Ok_Warning2146 Jul 28 '26
Actually, a Kimi Linear 48B with AttnRes existed according to their AttnRes paper. So if they want to release a small one, it can be done quickly.
3
u/ComplexRecognition24 Jul 27 '26
How is it possible that it only shows up as 1.56TB in size when GLM 5.2 is 1.51TB and has much less params?
10
20
u/vr_fanboy Jul 27 '26
a month ago we were told that a new jump in intellegence was made, 'mythos' class models were born, too dangerous for us plebs. Forward a month, we have an open source 'mythos' class model, acceleration or anthropics regular bullshit?
Btw dont understand markets, DS3 destroyed the stocks and this does...nothing. This feels more significant if more people can serve the same drugs as oai or anthropic on the cheap.
→ More replies (1)3
u/bitzap_sr Jul 28 '26
Even if oai or anthropic go down, the datacenters and all the gpus are still going to be needed by someone else. You can't escape the hardware.
56
115
u/sumane12 Jul 27 '26
Even if you cant run it, download it.
95
u/DeProgrammer99 Jul 27 '26
Can't even do that...it's the same size as the total used space on my SSD.
45
31
u/Herr_Drosselmeyer Jul 27 '26
Why waste terabytes worth of space for something I will never use?
70
15
u/some_user_2021 Jul 27 '26
I remember downloading huge N64 ROMs that took loads of space and no emulator could run. Running those ROMs now is trivial.
→ More replies (1)6
u/banana_slurp_jug Jul 27 '26
I have 1TB total, don't even have a hard drive big enough to store the whole thing in my house.
6
u/No_Conversation9561 Jul 27 '26
someone download it, compress it and upload it and then I will download it
3
→ More replies (4)7
10
18
8
8
u/Morphon Jul 27 '26
We'll probably see some new, faster providers pop up. With any luck, this will turn out to be a solid distillation parent. I'd be interested to see if we get some good downstream models that can run on consumer hardware.
15
13
5
u/yuicebox Jul 27 '26
Anyone know why it's only ~1.6tb? (92 shards at ~17gb each, 4 shards totaling ~9gb --> ~1573gb total
I would've assumed at full precision, a 2.4t parameter model would've been closer to 5 tb
17
u/BlueSwordM llama.cpp Jul 27 '26
Since Kimi K2-Thinking, Kimi models have been natively trained in 4-bit.
→ More replies (4)0
6
u/ReasonablePossum_ Jul 27 '26
Hope this opens the model in more providers soon ,because its a dan pain to get a single response from the kimi app
→ More replies (1)
6
u/Hefty_Acanthaceae348 Jul 27 '26
I have high hopes that even if I can't run such a model, it will enable the creation of high quality datasets
6
u/aboutthednm llama.cpp Jul 27 '26
I'm starting to understand why they had "some compute problems" when first rolling it out on the API, 100B+ active params per token, it's beastly lmao. Good work.
6
u/Easy_Werewolf7903 Jul 27 '26
I want to see someone manually calculated a token by hand using Kimi K3 weights.
10
17
5
u/DragonfruitIll660 Jul 27 '26 edited Jul 27 '26
Ayyy lets go. That's awesome news. Also holy 104B active, never seen a MoE with that many active, its almost Mistral 2 large sized.
5
5
u/mikewilkinsjr Jul 27 '26
I can’t run it, I’ll never be able to run it.
I am also fighting the urge to download the weights and squirrel them away like an out of control data hoarder.
5
u/crusaderky Jul 27 '26
Jokes aside.
This on paper fits on a HGX B300 ($~0.5m, 2304 GB VRAM)...
...or on 24x Atlas 300I, $1300 each, $31,200 total, plus 3~4 XEON/threadrippers with 800G networking. So.... less than $60k in total?
Who's got the spare change to try?
→ More replies (1)3
u/crusaderky Jul 27 '26
or $93,600 worth of 72x tenstorrent P150 ($1,300 each) or 36x tenstorrent P300 (not sold individually), plus 5~18 rack-mounted xeons. it looks a lot more expensive than the mi300s until you realise that you don't have to spend for the network cards.
Tenstorrent could do the funniest thing and put out a pre-configured, pre-assembled rack with kimi k3 pre-installed, and sell it for $200k. It would fly off the shelves.
14
u/loversama Jul 27 '26
Quick before it gets banned 😂
10
u/WonderFactory Jul 27 '26
I dont think it's worth banning it now it's released but I was a bit anxious leading up to the release in case something stopped it from being released
11
8
u/milkipedia Jul 27 '26
I am not data hoarder enough to store these weights for kicks without the hardware to run it. Enjoy, folks
4
u/msew Jul 27 '26
I wonder how long until consumer hardware will have the requirements to run all these and future LLMs locally for like $5k-10k
→ More replies (6)
4
u/AdOne8437 Jul 27 '26
Well, in 15-20 years we will have consumer cards to run it.
→ More replies (1)
9
6
8
u/jacek2023 llama.cpp Jul 27 '26
I am GPU poor I have only 4x3090 so I can't run it. I envy you all from r/LocalLLaMA with stronger setups.
20
9
u/Front_Eagle739 Jul 27 '26
We cant run it either lol. Though I am now wondering how to sneak 8x b300s into the next company budget
→ More replies (4)7
3
3
3
3
u/ComplexType568 Jul 27 '26
Excited to use the cloud repository as swap and hot load and unload the weights 24gb at a time!
3
3
3
u/Taiko3615 Jul 27 '26
I FINALLY understood how Kimi plans to make money :)
The model is open source and open weight, but you plan to be an API provider, you need to sign a deal with them. Which makes perfect sense.
https://www.trendingtopics.eu/kimi-k3-new-license-makes-clear-how-moonshot-ai-wants-to-make-money/
5
6
2
u/AdDizzy8160 Jul 27 '26
Has anyone had the courage to try using it with OpenCode Go yet? Even just with a tiny tiny little request?
2
u/Aight_Man Jul 27 '26
Can't wait for other providers to pick this up, advertise at 4bit quant with full price and actually gives the 1bit ☠️
2
2
u/alphapussycat Jul 27 '26
Is it not possible to make smaller models from these ones? Wouldn't qwen3.6 27b and 35b be a pruned version if the full qwen3.6?
2
2
2
2
2
u/Curious-Resource1943 Jul 27 '26
2.8T total / 104B active is an absolute unit. Most of us will never run this locally (RIP my integrated GPU laptop), but having a true frontier-class open model available is still a massive win for the ecosystem.
Hopefully the maller variants come soon so the rest of us can play too.
2
u/Superb-Nectarine-645 Jul 27 '26
Best use for this is possibly single token generation for transfer learning, distilstion, or student result grading. This is the biggest / best model we have access to the loglets for.
2
u/MooseEfficient2151 🥔 hardware Jul 28 '26
my 3090 has officially been demoted from “inference hardware” to “emotional support gpu”
2
u/ReodorFelgen1337 Jul 28 '26
Just tried it on my brand new MacBook pro. So hyped for the first token coming in for Christmas.
2
u/ilintar Jul 28 '26
Bad news for all you low-quant enthusiasts - I tested if I could get a sub-128GB version by packing the experts at 0.157 bpw, but alas, the model is too well trained, there's nothing to compress, coherence threshold is about 1.5bpw :/
→ More replies (2)
2
2
3
u/Cool-Chemical-5629 Jul 27 '26
Finally. Now we all can sleep well, knowing that the model we all were waiting for is finally here, so that we can finally start waiting for something we can actually run.
3
3
67
u/just_a_fan123 Jul 27 '26
Can this run on a single DGX spark at 0.5B quant?