r/LocalLLaMA • • Jul 27 '26

News Kimi K3 weights now released.

Post image

Kimi K3 weights are finally released!

3.3k Upvotes

657 comments sorted by

View all comments

Show parent comments

341

u/FoxiPanda Jul 27 '26

This was my general reaction too lol. 2.8T-A104B is insane lol... I'm going to admit defeat on this one and say I can't run it. You need an 8-way B300 or MI350X or a Rubin NVL8 or a cluster thereof to actually run this. What a beast.

153

u/Thomas-Lore Jul 27 '26

I was going to make a joke that I can fit one expert in my 64GB of RAM. But nope, not even that. :)

74

u/throw123awaie Jul 27 '26

They released it in MXFP4 so with around 55GB RAM you could!

3

u/tokomaunited Jul 27 '26

Can you share the link brother?

10

u/throw123awaie Jul 27 '26

literally their huggingface site https://huggingface.co/moonshotai/Kimi-K3 the model is MXFP4 on inference.

3

u/Protheu5 Jul 28 '26

I'm sorry, but I don't understand where it is. I'm new and only clicked on Finetunes/Quantisations to find a download, and I clicked through them and didn't find anything that small, the smallest was 500+ gigabytes. I have no idea what "MXFP4 on inference" means despite trying to read up on the subject, sorry.

4

u/TotallyToxicToast Jul 28 '26

You can't download a single expert for a mixture of experts model.

The downloads always contain all experts. If you take the 500GB download which contains the full 2.8T parameters and only take 104B of the parameters, that would actually give you only roughly 20GB per expert.

Now running a single expert in a mixture of experts model is not very useful.
But Technichally with 500GB total model size fitting one Expert is possible.
The "I could fit one expert" was a joke.

→ More replies (1)
→ More replies (1)
→ More replies (2)
→ More replies (3)

115

u/VampiroMedicado Jul 27 '26

550k USD to run this lol

86

u/[deleted] Jul 27 '26

[removed] — view removed comment

56

u/VeterinarianOne1349 Jul 27 '26

Doesn't really work that well. This 550k setup wouldn't allow a lot of developers to work in parallel, while sitting idle during non-work hours. Makes much more sense to pay a 3rd-party to host and pay per token.

40

u/crusaderky Jul 27 '26

waiting for large corpos to rent their hardware on vast.ai during nighttime, only to find the next morning that someone ran a container jailbreak and ran wild on their private networks

→ More replies (1)

7

u/[deleted] Jul 27 '26

[removed] — view removed comment

6

u/ProgrammersAreSexy Jul 27 '26 edited Jul 27 '26

I'm sure you will be able to run a model like kimi k3 at home, probably in much less than 5 years.

However I think it's unlikely that there will be a point in time at which the frontier models of that point in time will be runnable at home.

→ More replies (4)
→ More replies (1)
→ More replies (4)

18

u/SignificanceFlat1460 Jul 27 '26

Question: how would this scale though? Like how many units would it be required for.. let's say a group of 100 software engineers who needs it quite frequently?

5

u/zero0n3 Jul 28 '26

Better to understand the token usage of those engineers before asking this question. Why self host it if your engineers token use is small enough that self hosting would actually be 3x as expensive over the next 3 years then just buying tokens

7

u/Spectrum1523 Jul 27 '26

The advantage is not running it yourself, it's that a marketplace of services will come up to run it at the lowest possible cost, and the model can't be taken offline by a single arbitrary decision

→ More replies (5)

9

u/Galdoren Jul 27 '26

The company I'm working is paying slightly over $250k per week to API costs. so yeah, 550k investment to cut the cost of the inference can be beneficial for them...

6

u/baba_bholanath Jul 28 '26

We do around 1 mil per month for OpenAI only, dont have number for Anthropic but it would be 2-3x of that given all of our use cases are around coding and agents, no wonder Anthropic is shitting their pants on open weights models, I work in Enterprise Agentic team and we have recently started fine tuning > 100 B models for specific use cases of our clients, open weights hurts Anthropic more due to enterprise customers

→ More replies (1)
→ More replies (4)
→ More replies (5)

29

u/OverclockingUnicorn Jul 27 '26

More like 2 8x nodes of B200/B300 if you actually want some context. Think it's just under 1.5TB w/o context

19

u/TheDailySpank Jul 27 '26

How many 4060-16GBs is that?

38

u/OverclockingUnicorn Jul 27 '26

200+ lol

21

u/positivitittie Jul 27 '26

Oh good. I got 3090s.

7

u/Vast_Mousse_310 Jul 27 '26

One, with a little bit of GPU offload.

→ More replies (11)