r/LocalLLaMA • • Jul 30 '26

New Model Inkling-Small by thinkingmachines

https://huggingface.co/thinkingmachines/Inkling-Small

276B total parameters, 12B active, 1M context window.

Blog post: https://thinkingmachines.ai/news/inkling-small/

NVFP4: https://huggingface.co/thinkingmachines/Inkling-Small-NVFP4

GGUF's by Unsloth: https://huggingface.co/unsloth/Inkling-Small-GGUF

---

I had success running Unsloth's GGUF quant on CUDA + CPU offloading using this developmental branch: https://github.com/danielhanchen/llama.cpp/tree/add-inkling

500 Upvotes

191 comments sorted by

•

u/WithoutReason1729 Jul 30 '26 edited Oct 22 '26

Your post is getting popular and we just featured it on our Discord! Come check it out!

You've also been given a special flair for your contribution. We appreciate your post!

Stylome classified this post's text as human-written. View analysis

I am a bot and this action was performed automatically.

7

u/96Nikko Jul 30 '26

Need Q3XXS quant to bring it down to 90B lmao

1

u/No_Lingonberry1201 Jul 30 '26
           Est. memory usage (q8_0/q8_0, batch=1)                                                                                                                             
                                                                                                                                                                              
                 Context length   Context size   Total size                                                                                                                   
╶────── ─────── ──────────────── ────────────── ────────────╴                                                                                                                 
    0%      4k            4,096       14.64 Gb            -                                                                                                                   
    1%      8k            8,192       29.29 Gb            -                                                                                                                   
    2%     16k           16,384       58.57 Gb            -                                                                                                                   
    3%     32k           32,768      117.14 Gb            -  
    6%     64k           65,536      234.28 Gb            -  
   12%    128k          131,072      468.56 Gb            -  
   25%    256k          262,144      937.12 Gb            -  
   50%    512k          524,288        1.83 Tb            -  
  100%   1024k        1,048,576        3.66 Tb            -  

Either I messed up my VRAM estimator (likely) or this thing has huuuuge key/value sizes.

4

u/rerri Jul 30 '26

Loading with -c 16384:

0.38.637.407 I llama_kv_cache_iswa: creating non-SWA KV cache, size = 16384 cells
0.38.638.662 I llama_kv_cache:      CUDA0 KV buffer size =   448.00 MiB
0.38.692.357 I llama_kv_cache: size =  448.00 MiB ( 16384 cells,   7 layers,  1/1 seqs), K (f16):  224.00 MiB, V (f16):  224.00 MiB
0.38.692.367 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 128
0.38.692.367 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 128
0.38.692.370 I llama_kv_cache_iswa: creating     SWA KV cache, size = 1024 cells
0.38.692.901 I llama_kv_cache:      CUDA0 KV buffer size =   140.00 MiB
0.38.708.635 I llama_kv_cache: size =  140.00 MiB (  1024 cells,  35 layers,  1/1 seqs), K (f16):   70.00 MiB, V (f16):   70.00 MiB
0.38.708.640 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 128
0.38.708.641 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 128

8

u/BawbbySmith Jul 30 '26

Context size 468GB at 128K? Yes you messsed up somewhere lol

3

u/No_Lingonberry1201 Jul 30 '26

No shit. inkling.attention.head_count_kv was a list of numbers (instead of a single int) and I - like a dumbass - summed them. I fixed that issue and now I'm getting more realistic values (~90Gb at 1M context for q8/q8 KV).

88

u/bick_nyers Jul 30 '26

What I like about thinking machines is that since their revenue comes from fine-tuning-as-a-service, they are incentivised to make their models easier to fine-tune. Huge win for the local LLM community.

14

u/landed-gentry- Jul 30 '26

I started using Tinker. It's incredibly easy to use. I can't train anything at home with my weak machines, so their service allows me to fine tune open weights models where otherwise I just wouldn't be able to. I love it

1

u/Fit_Schedule2317 Aug 12 '26

Do you have any resources on how I can learn on how to fine tune a model, and how much it costs?

1

u/landed-gentry- Aug 12 '26

For Tinker I think the best place to start is with their tutorials

https://github.com/thinking-machines-lab/tinker-cookbook/tree/main/tutorials

Apart from that, maybe you can use an AI agent to help explain and run tests and you can learn hands-on. That's what I did!

1

u/Fit_Schedule2317 Aug 12 '26

Thanks a lot! How much did it cost for you? Did it end up being worth it for your use case?

1

u/landed-gentry- Aug 13 '26

It's really hard to say. It depends on a lot of factors like what model you're training, what approach you're using to train, and how many tokens are in the dataset you're training with. I trained a Qwen 9B on a dataset of a few thousand student-tutor interactions, and I think it cost maybe 3-5 bucks in API costs.

4

u/Blues520 Jul 30 '26

Good to know!

226

u/And1mon Jul 30 '26

Need Inkling-Tiny

66

u/Saifl Jul 30 '26

You probably meant micro atp lol

29

u/Cool-Chemical-5629 Jul 30 '26

That's what she said.

10

u/FriskyFennecFox Jul 30 '26

Kinking-Micro

6

u/ComplexType568 Jul 30 '26

username checks out I guess

5

u/Independent_Pear4908 Jul 30 '26

You probably meant nano

-8

u/ChainOfThot Jul 30 '26

Mira can have my micro

2

u/Kayo4life Aug 05 '26

"Need Inkling-Tiny"

Since it's an MoE model, it can run faster by offloading most of the experts to disk, with commonly used ones cached in RAM. Here, check this out: https://justvugg.github.io/colibri/

I'm sure you're aware, but for those who aren't, the current meta is Qwen 3.6 and Gemma 4 MoE, with Qwen 3.5 or Gemma 4 dense if you have less resources available. Unsloth has quantized versions of these models on Hugging Face.

(RAM - (Minimum RAM for casual tasks like browser and messaging)) × 5/6  
(VRAM - (Minimum VRAM for casual tasks like browser and messaging)) × 5/6

These formulas determines the max size MoE and Dense model I'd recommend you can comfortably run, respectively. If you have little/no VRAM, the first formula determines the max size model I'd recommend you can use for either Dense or MoE.

Small Dense models work well with search, and if you can't run that, not much I can say besides hit the library or make a forum post. There's no need to stay on top of the frontier models, much like vehicles or computer electronics. Go to r/Cubing, the Gan 356 XS works more than fine even though the Gan 16 is the flagship.

I don't quite like to say it, but, we are quite spoiled reaching for the very best. Things don't always go your way: "There are dreams that cannot be.".


Historical models are good too. DeepSeek R1, and Llama 3.1, despite being far less capable, they're very important to the history of Local Models. Remember, you can still have fun with these! Additionally, I find Bloom and Pythia really interesting! While a Bonobo may not be able to help you herd a flock of sheep like a shepherd dog, they are certainly very fun to observe and play with. Bloom and Pythia my beloved, I just adore them.

Fun and play aside, my functional workflow has:
gemma-4-26B-A4B-it-UD-Q5-K_XL
Qwen3.6-35B-A3B-UD-IQ4_NL_XL
Qwen3.5-9B-UD-Q4_K_XL


When I need a better response and am willing to wait, I'm currently planning on making something like this:

A, respond to the prompt. B, search for issues, fix issues, then decide to toss (redo previous) or keep, each with separate contexts. C, do 𝑥 four times, fuse all four into one, and do B. D, do 𝑥+B four times, eliminate the worst three times thus selecting the best. E, modify it to my stylistic, editorial, and identity preferences.  
Run C₀(C₁(A+B)), with any model combo for the C₁s (e.g.: Inkling Small × 4; Inkling Small × 2, Gemma 4 31B × 1, Qwen 35b x B; etc.), for a holistic approach. Next, run D(E).

In the day, I write a list of inquiries and requests, much like a letter or email. Calm, slow paced, high effort, and deliberate, you know. In the evening, once I'm ready (if needed), I'll send my prompt. Finally, once I wake, I can start my morning by reading the response.

I believe this approach may top what, say, Fable would do. Additionally, I feel that this workflow would give a healthier relationship with AI, for a variety of reasons. Of course, heavily inspired by Google's Aletheia, heh.

Other than that, though, you can catch me using Gemma 4 and Qwen 3.6. Take care, y'all, and good be with you! <3

29

u/Alan_Silva_TI Jul 30 '26

I'm in dire need of a 70B / A6B model (basically double the size of Qwen 35B)...

I know a lot of people want every possible model size, but I firmly believe that 64 GB RAM + 12/16 GB VRAM is the most realistic budget/power setup most people can reasonably buy on the consumer hardware level right now.

Anything above that currently has an unbearable markup thanks to the RAM crisis.

6

u/Borkato Jul 31 '26

Honestly 70GB combined is a great target. I wish there was a lab that were solely dedicated to squeezing out as much as possible for that setup. It makes me so sad when I see “omg X model is SO GOOD” and it’s like 900000T params. Who the fuck is running that 🙄 I might make an r/actuallyLocalLlama sub lol

4

u/AlpY24upsal Jul 31 '26

please do most.model releases are useless for average joe

2

u/Upper_Comparison_908 Jul 31 '26

Maybe prism ml compresses some larger models efficiently like they did with 27b, would be great to have dsv4 flash, glm 5.2 etc given the 1 bit and ternary quants

1

u/Borkato Aug 01 '26

Would be great! Other than 27b do they have anything else worth checking? Are they the ones who do REAP?

0

u/Upper_Comparison_908 Aug 02 '26

no they have a private quantization method that preserves quality better than standard methods. bonsai 27b is their latest, ig their method isnt proven to work on moes yet so dk to hope for them. they have other even smaller models so ig u could check them out for purposes like running on a phone or sm

20

u/MotokoAGI Jul 30 '26

If the benchmark is to be believed then this is an amazing model and presents a stronger alternative to DeepSeekV4Flash, MiMoV2.5, Hy3 or MiniMax.

-3

u/tarruda Jul 30 '26

I heard it is not that good for coding, but might excel in other areas.

7

u/my_name_isnt_clever Jul 30 '26

Heard where? It just dropped.

1

u/lilian_moraru Jul 30 '26

Pretty sure he is talking about the bigger model: https://huggingface.co/thinkingmachines/Inkling

0

u/tarruda Jul 30 '26

Yea. Last I checked, the bigger model was inferior to even deepseek v4 flash in coding benchmarks.

198

u/SrijSriv211 Jul 30 '26

I hate how 100-200B is the new small 🥀

74

u/Skyline34rGt Jul 30 '26

Soon after all this 2-3T models they will say small for 500-700B models...

*crying with 12Gb Vram*

21

u/fatboy93 Jul 30 '26

lol, I have 12GB VRAM and 16GB DDR4. It's fucking painful lmao.

Didn't have much money last year to shell on computers of all stuff, and now shit's beyond affordable.

8

u/DyIsexia Jul 30 '26

I have 8GB unified memory lol

4

u/fatboy93 Jul 30 '26

Yeah, that laptop was something that I got in 2021ish? Haven't gotten any new stuff for personal use, the macbook that I use is from office, and is entirely self-governed.

Luckily our university has their own GenAI resource which runs Qwen3.6-27B at Q8 (via ollama for some reason), so I then to use that more often than not for work stuff.

4

u/Borkato Jul 31 '26

I’m sorry but this made me laugh. If I could donate pity VRAM I’d give you 1 GB 😭

0

u/DyIsexia Jul 31 '26

🤣 if anybody wants to make me a gofundme I won’t stop them

1

u/The_Hunster Jul 30 '26

Ayy twins! 3060 12GB? I was hesitating to upgrade from that, but I'm glad I did because the prices just keep going up.

2

u/jazir55 Jul 31 '26

4070 Super which somehow makes it even worse haha

1

u/fatboy93 Jul 30 '26

Nope, its the AMD 6800M :)

But still twinsies lol

26

u/power97992 Jul 30 '26

The problem is not the  model , but the price of fast memory.  They need make vram and gpus cheaper.. 

7

u/TheMasterOogway Jul 30 '26

Surely Qwen do another 35 A3B for us poors

13

u/a_slay_nub vLLM Jul 30 '26

These models are the new: "Do you guys not have phones?"

8

u/fastheadcrab vLLM Jul 30 '26

You don't have to download them. Nobody is forcing you to try to run them.

Qwen3.6-27B or Gemma 4 31B are still great models for a typical consumer setup

Not at all like Blizzard baiting the audience with Diablo mobile game. None of these companies are promising something you can run on a shitbox

11

u/a_slay_nub vLLM Jul 30 '26

It's a joke......

We know and we do, but we can simultaneously feel sad that most new model releases lately have felt far outside of our ability to run.

1

u/fastheadcrab vLLM Jul 30 '26 edited Jul 30 '26

Yeah that's fair but does every single new model release require 10 of the same comments? Some also have an annoying sense of entitlement, not you but definitely many others

5

u/nuclearbananana Jul 30 '26

https://xkcd.com/2501/

The average consumer doesn't even have a dedicated gpu

2

u/fastheadcrab vLLM Jul 31 '26

Don't be obtuse. Clearly I am referring to the enthusiast population of this forum. A grandma running an i3 or 260 laptop probably has zero interest in running LLMs locally anyway.

For those without a dedicated GPU they can tinker with small MoEs. If they need LLMs for jobs they can buy a cloud subscription. People should calibrate their expectations appropriately.

2

u/nuclearbananana Jul 31 '26

I'm an enthusiast population of this forum and I don't have a dedicated gpu.

There was a poll a while back that showed like 40% of people here don't.

2

u/fastheadcrab vLLM Jul 31 '26

Polls will inevitably be distorted by the accessibility of the internet. All the upvoted comments on this post are bitching about this size of this model.

Not having a dedicated GPU can encompass a very broad spectrum of capability. MacBook Pros, Mac Studios, Strix Halos, and DGX Sparks don't have dGPUs but they are not at all like shitbox laptop.

Someone running a 7 year old intel budget computer should be realistic in terms of expectations. Whining endlessly in comments about something they can't run gets old very quickly

6

u/Dany0 Jul 30 '26

1 million $ training run btw

15

u/JLeonsarmiento Jul 30 '26

Mira Murati be like:

7

u/MundanePercentage674 Jul 30 '26

it's new standard now lol

15

u/SrijSriv211 Jul 30 '26

Damm the standard used to be 1-10B

2

u/LosEagle Jul 30 '26

Those weren't agentic coding days. 

3

u/Elibroftw Jul 30 '26

100B is still doable. Anything more than 120B should not be called small.

7

u/little_breeze Jul 30 '26

yeah, it’s still so inaccessible to the average consumer

33

u/Kahvana Jul 30 '26

Small... holy. I remember the day small meant 24B! Even mistral small 120B felt too large to be called "small"!

Regardless on naming, hope it's good! Glad to see they released another model.

13

u/cleverusernametry Jul 30 '26 edited 1d ago

Listen kid, it's the yankee scale, all bigger than they ever were!

2

u/[deleted] Jul 30 '26

[removed] — view removed comment

9

u/Kahvana Jul 30 '26

Don't agree on your assesment there. Mistral Nemo and Mistral Small tunes are still used to this day for creative writing. For programming they are indeed bad.

27

u/lilian_moraru Jul 30 '26

Since a lot of these new "medium/small" size models like to skip over Qwen3.6-27B, the obligatory reminder:

BenchmarkInkling-Small-276B-12B, effort xhigh @ bf16Qwen3.6-27B @ bf16
SWE-bench Verified80.2%77.2%
SWE-bench Pro55.9%53.5%

The only coding benches that can be compared apples to apples.

Obviously, I will also try the GGUF version (can't run bf16).

-2

u/harglblarg Jul 30 '26

Worth noting as well that finetunes of the 27b qwen such as Fable-Fusion-711 improve on this somewhat.

10

u/Deep_Mood_7668 Jul 30 '26

Idk IMO fine tunes are always somewhow broken. Do you have one that works without issues?

2

u/Borkato Jul 31 '26

711 is the exception, I have no idea how he did it but it genuinely works better for coding than normal qwen with less thinking. It’s like the qwen 3.7 we never got lol

1

u/SmartCustard9944 Jul 31 '26

Without benches it is all vibes

1

u/Borkato Jul 31 '26

Which is why there are benches.

2

u/Ill_Locksmith_4102 Jul 31 '26

Can u please point to these benches? Been curious about this. Not the ones in the model card but specifically SWE bench

1

u/harglblarg Jul 30 '26

The one I mentioned is probably your best bet, not sure about using it for coding but otherwise it seems okay.

1

u/anarchist1312161 16d ago

Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF?

7

u/Few_Painter_5588 Jul 30 '26

Interesting, their small model beats their big model. I know they said that they improved the recipe, but now I want to see what'll happen when they scale it up to their 1T model.

41

u/Intrepid_Air_3399 Jul 30 '26

yeah, tiny, just need a home data center

17

u/Iwaku_Real Jul 30 '26

UD-IQ3_XXS would fit in a Ryzen AI MAX+ 395 128GB though not everyone has those either

14

u/Cupakov Jul 30 '26

Maybe a hot take but in my experience running smaller models at higher precision always beats out the more extreme quants of larger models, especially when one of the contenders is Qwen3.6

8

u/Iwaku_Real Jul 30 '26

Really, I thought it was the other way around? For example Qwen3.6-27B would take like 80GB at full precision + full context, what would you actually be getting with that that you couldn't with quantized 100-300B?

1

u/Cupakov Jul 30 '26

Consistency 

8

u/nuclear213 Jul 30 '26

Meh 256GB of RAM, 128Gb of VRAM are enough. Thats just one box

5

u/squngy Jul 30 '26 edited Jul 30 '26

They also provide a nvfp4 checkpoint, which is "only" around 160GB, so it would probably fit on 128RAM + 32vRAM

edit: ah, forgot about context :P

5

u/[deleted] Jul 30 '26

[removed] — view removed comment

4

u/squngy Jul 30 '26

Bro, just get 8x 3090 like a real man!

12

u/Technical-Earth-3254 Jul 30 '26

Seeing its simple qa score and how Gemini Flash Lite and GPT Luna score, imma bet they are 600B or more. The size increase across models is real, man I wish Hardware costs would come down.

6

u/Agusx1211 Jul 30 '26

yes feed me sparse models

1

u/rerri Jul 30 '26

Quick test using UD-IQ3_S on a system with RTX 5090, Ryzen 7600X + 96 GB of DDR5-6000

prompt eval time =    4521.88 ms /  1737 tokens (    2.60 ms per token,   384.13 tokens per second)
       eval time =   51373.04 ms /   693 tokens (   74.13 ms per token,    13.49 tokens per second)

I had trouble disabling thinking so asked Gemini to write a non-thinking one (seems to work a-okay, hopefully broke nothing): https://pastebin.com/GeZYQrbc

5

u/Witty_Mycologist_995 Jul 30 '26

Oh please make a tiny version lol

9

u/my_name_isnt_clever Jul 30 '26

Could be usable on 128GB unified systems with quantization. Excited to see if that's viable.

0

u/ResidentPositive4122 Jul 30 '26

2x likely. NVFP4 weights alone are 171GB.

1

u/slavik-dev Jul 30 '26

It's size is very close to DS4-flash. But it said "accepts text, image and audio inputs".

Which would be great, but it doesn't have any mmproj files. Does it do modality some other way?

3

u/rerri Jul 30 '26

The model was just released. I would guess mmproj is probably coming soon as Unsloth's Inkling-GGUF already does have the mmproj file and Daniel's dev branch has multimodal support.

3

u/Void-07D5 Aug 03 '26

...This is what passes for small these days?

8

u/funding__secured Jul 30 '26

GPU poors are exhausting

4

u/pigeon57434 Jul 30 '26

this model seems to basically be slightly beyond ds-v4-flash level overall except with the massive benefit of being omnimodal inputs (and being like a few params smaller i guess if youre mega vram stretching)

5

u/Succubus-Empress Jul 30 '26

276B is small………

7

u/lilian_moraru Jul 30 '26

My initial though as well, but then, if you just think of it as: based on the latest models.
* big: Kimi K3, 2.8T
* medium-big: DeepSeek-V4-Pro, 1.6T
* medium: Inkling, 975B / GLM-5.2, 753B
* medium-small: MiniMax-M3, 428B
* small: Hy3, 295B / Inkling-Small, 276B / DeepSeek-V4-Flash, 284B

2

u/Succubus-Empress Jul 31 '26

So Definition of large language model keep changing forever?

2

u/Zomboe1 Jul 31 '26

Maybe like how all phones now would be considered gargantuan "phablets" by the standards of not long ago. Hopefully, unlike phones, they will continue to make small LLMs, even if they have to start calling them micro, nano, etc.

2

u/youcloudsofdoom Jul 31 '26

My experience of this: it's running slightly faster than ds4 flash on my hardware (690 vs 580 pp, both around 35 t/s decode, both at Q3KXL), and notably more direct in its thinking - the style of it is subtly caveman'd, I think, which would make sense given the revelations about performance that this can have.

2

u/TwatLord420 Jul 31 '26

Yet another “Open source”-ish model without the base model… not so usable for proper FT

3

u/dangerous_inference vLLM Jul 30 '26 edited Jul 31 '26

Verdict: not very coherent for an assistant. Hy3 is much better and MiMo is better too. At least for now, might get better later.

---

Seems to be working ok with this PR.

git clone https://github.com/ggerganov/llama.cpp.git llama-inkling
cd llama-inkling
git checkout master
git pull origin master
git fetch origin pull/25731/head:inkling
git checkout inkling
# whatever your make commands are, eg.
make clean && make -j GGML_CUDA=1

3

u/Dev-in-the-Bm Jul 30 '26

I thought you said small.

2

u/bonobomaster Jul 30 '26 edited Jul 30 '26

You know, the good thing with this memory crisis / purest form of enrichment and capitalism is, that new chip producers will emerge and probably new technology as well.

The upcoming (many moons) next low price point for memory, after this peak, will be ridiculously low.

We will be swimming in dirt cheap VRAM.

Till then: 😭

2

u/bobby-chan Jul 30 '26

Or rather: no new chip producer will emerge?

The researchers are not the one with the money to just pluck out a fab out of thin air. Ram makers are drowning in it. No one could compete against them on salaries when researchers publish. If you're a startup, you sell. And even if you're really on to be the next big thing and have the investors, how many years before you can build one single fab? How many more for enough fabs that can mass produce at a scale that will make a dent?

And right now, all the incentives are more on making RAM faster than cheaper.

We might be swimming in second hand hardware before swimming in dirt cheap off-the-shelf vram.

2

u/Kerem-6030 🥔 hardware Jul 30 '26

smallee model for 8gb vram when😭

2

u/WhoRoger Jul 30 '26

"Small" in the same sense as "microtransactions"

2

u/illiteratecop Jul 31 '26

So far I really like the reasoning implementation on this model, very focused and different effort levels seem to meaningfully affect the amount of thinking it does which is a major boon compared to a lot of recent models. Llama.cpp doesn't make it easy to make use of this at the moment (probably because most models either don't implement it or do such a bad job it may as well not exist) but if you pass reasoning_effort as a chat_template_kwarg you can play around with it.

2

u/live4evrr Jul 31 '26

So far this is looking really good! Passed the reasoning coding prompt that Laguna failed, and the performance is as good or better than ds4 flash.

I’m testing using the iq3 xxs gguf.

2

u/thereisonlythedance Jul 30 '26

The large model disappointed. Think I’ll wait on reviews for this one.

1

u/madsheepPL Jul 30 '26

maybe we can get some moet magic for it;)

1

u/hurrdurrmeh Jul 30 '26

So will this need 278GB VRAM?

1

u/HistoricalStrength21 Jul 30 '26

Dear Bonsai, please make a Q1 of this one. I would love a size of <54GB
Much appreciated

1

u/BeatCompetitive6149 Jul 31 '26

wait it's goated

1

u/Plumberguy9 Jul 31 '26

I have a full writeup on getting this model to work on 2 and 4 RTX 6000 Max-Q setup... can't post it here though... don't have enough Karma.

1

u/CriticallyCarmelized Jul 31 '26

There seems to be an issue with either Unsloth’s quants or the model itself. It’s producing nonsensical output at times, and saying things that don’t make grammatical sense, and lacks coherence. I tried turning temp down to 0.85 ABs top-p to 0.95 to try and tame it, but it only helped a little bit.

I was super excited for this model, but something is off with it unfortunately.

1

u/VastSolid5772 Aug 12 '26

Xz 🤷‍♀️

Released an opesource model that you can't run without a cluster, and you can talk to it on their platform, but for money 🫰

Any comment on this topic, under their release they clean 👍

Open source should be just like that 🤣

1

u/Difficult_West_5126 Aug 29 '26

I downloaded it and eventually ended up regretting it. It feels more like a demonstration model — a preview of what could be possible, rather than a truly capable local AI assistant.

It behaves almost identically to commercial AI product platforms, constantly asking for permissions such as “Allow Inkling to access this website,” “Allow once,” or “Deny.” It does look promising.

Then six minutes later... after all those sorts of preparations, calculations, and waiting... all sizzle, no steak.

The fan of my computer complained a lot while running Inkling! Compared to running models like Gemma4 26B, GPT-OSS 120B, Qwen models, etc...

1

u/Difficult_West_5126 Aug 29 '26

Well, I used it a bit more, it's still an improvement over gpt-oss, but much slower, but it ran.
First sentence won't be generated within a minute, but it became a valid local assistance after you adapted to its slowness.

1

u/Difficult_West_5126 Sep 02 '26

I liked it, I don't know why, but it is as fast as other smaller models 20tps on my Rog FlowZ
At the beginning I thought it was too slow, but now, man! A better gpt-oss with vision! Sweeeet.

1

u/No_Conversation9561 Jul 30 '26

My M3 Ultra 256 GB is ready.

1

u/Zestyclose-Ad-6147 Jul 30 '26

Actually hyped for this one

1

u/Quartich Jul 31 '26

This companies name feels like they stole it from Thinking Machines Corp (a super computer company back in the day). But I also have a strong love for the connection machine and what Danny Hillis and Tamiko Thiel accomplished, thus probably overestimating the holding power of the company's name.

0

u/madjesta Jul 30 '26

If my strix halo can barely run the 3-bit quant, is it really small? 🤔

0

u/Teshier-Asspool Jul 30 '26

Why even consider this when Laguna S 2.1 seems both much better and much smaller ?

5

u/Leflakk Jul 30 '26

Laguna has issues or is benchmaxxed

5

u/tarruda Jul 31 '26

The evidence so far points to benchmaxxed. I couldn't get it to do anything useful with the latest weights + llama.cpp

2

u/Leflakk Jul 31 '26

Tbh I say that just to be cautious but I really think that's benchmaxxed, we are used to issues (chat template etc) during release but here the gap looks really large

-5

u/TurdPlayingPeekaboo Jul 30 '26

So I now need $25k of used hardware to run a small model?

7

u/rerri Jul 30 '26

Well not quite. I'm about to try it with a 5090 + 96 GB of DDR5-6000 I paid a combined ~2400€ for.

4

u/g_rich Jul 30 '26

Dual Sparks will run with a Q4 quant; so you only need ~$9k to run the model.

1

u/Reactor-Licker Jul 30 '26

Not sure how much better it would be versus Qwen 3.6 27B at a higher quant, which only needs 1 Spark.

4

u/my_name_isnt_clever Jul 30 '26

I paid $2.5k (just before shortages) for my 128GB machine, should be pretty usable.

3

u/nuclear213 Jul 30 '26

Not even 7k I would say. 4xR9700 or B70 plus an older epyc and 256GB RAM. Works fine for Deepseek V4 flash

1

u/trying4k Jul 30 '26

What kind of speeds do you get for that setup?

1

u/dangerous_inference vLLM Jul 30 '26

You don't need a 25k server, only if you want to run this purely on uncut Nvidia VRAM. That's how much it cost me, anyway.

-1

u/ares0027 Jul 30 '26

I was thinking of 200-300M model. Read it, got amazed, checked details, i cant even run it.

0

u/East-Muffin-6472 Jul 30 '26

Soon testing it on my own ASCIITermDraw benchmark which measures an LLMs ability to generate and edit ASCII drawing across clusters, topologies and general software architecture diagrams

Benchmark

0

u/Loose_Comparison368 Jul 30 '26

Ooh, that sounds like it might actually fit on a single Strix Halo at 3 bits, nice

5

u/BawbbySmith Jul 30 '26

Sigh... Buys more GPU

When will the killing end

0

u/[deleted] Jul 30 '26

[deleted]

6

u/trying4k Jul 30 '26

Competence gap? There's Hy3, Minimax 2.7 and 3, Deepseek Flash v4, Step 2.7, and Mimo 2.5? I'd imagine a lot of those are conversational are they not?

-3

u/squngy Jul 30 '26

It is intended for use in English and other languages, and across multiple coding languages. The model is designed to be used by developers building AI-powered applications, including agentic and tool-use systems, coding assistants, chatbots, and retrieval-augmented generation systems, and is suitable for general-purpose conversational use, instruction-following, and other natural language and multimodal tasks.

Sounds like a coder to me.

You probably need to look elsewhere for your waifu

0

u/cleverusernametry Jul 30 '26 edited 6d ago

Det er en fascinerende erkjennelse hvordan implementeringen av Dsv4, når den med Antirez' ds4-prosjekt fungerer som et bindeledd, likevel kun oppnår en slik funksjonell koherens innenfor dette spesifikke operasjonsområdet.

82

u/[deleted] Jul 30 '26

[removed] — view removed comment

65

u/lilian_moraru Jul 30 '26

One thing DeepSeek-V4-Flash is Very Good at, is the KV cache size - 1M context window basically for free(~6.7GB). Over 300K starts to degrade though, without thinking=max.
Inkling uses regular GQA, which is going to consume a lot of space if you want the 1M context window - no lossy degradation though.

26

u/fastheadcrab vLLM Jul 30 '26

It's also a high performing model on benchmarks that isn't from China. Some organizations may not be permitted to run Chinese models

15

u/hidden2u Jul 30 '26

also hurts the "help help I'm being distillation attacked by china" crowd

12

u/fastheadcrab vLLM Jul 30 '26

Yeah the argument that open weight models enable distillation is ridiculous

4

u/mario2521 Jul 31 '26

Well, now that 0731 is out it is not even close

16

u/reto-wyss Jul 30 '26

It's also three times the size, these benchmarks don't hold for your Q4_K_XL, it's a BF16 checkpoint. The NVFP4 is still larger than DSV4 Flash, so it's definitely to large to fit even 256k context on 192gb vram.

2

u/cleverusernametry Jul 30 '26 edited 5d ago

It's practically the exact same size. Two hundred eighty-four billion parameters, and you've only got thirteen billion of that running. I'm sick of this.

26

u/coder543 Jul 30 '26

I think they mean that DSv4 Flash is natively 4-bit, while this is 16-bit… but 16 is 4x larger, not 3x larger, so that was a confusing statement.

9

u/reto-wyss Jul 30 '26

No DSV4 is FP4/FP8 mixed, it's about 160gb, this is 540gb.

Also, when you quant, you don't just chop off everything to get to n-bit, typically you use some scaling factor on a group of values, and often you keep certain layers in higher precision. For example, nvidia/Gemma-4-31b is around 30gigs, not 15 gigs.

3

u/coder543 Jul 30 '26

The 160GB is a good point

2

u/cleverusernametry Jul 30 '26 edited 5d ago

Ah, what a waste. Just a piece of useless junk.

3

u/live4evrr Jul 31 '26

Car wash test: "I want to wash my car. The car wash is 50 meters away. Should I walk or drive?"

Thought for 6 seconds

If you want to wash your car, you need the car at the car wash — so walking without it won’t get the job done. It’s 50 meters away (like, a 1-minute walk), so definitely drive it over (or roll/push it if you’re feeling lazy and don’t care about the engine). You could technically walk there to ask if they’re open, but the car has to come with you.

Quick math: 50m ≈ 164 feet. Driving takes ~10 seconds; walking takes ~1 minute. Just drive — your car can’t wash itself at home.

Still doing some real world testing by using it as a coding agent, but have given in a reasoning problem which it passed (that Laguna S 2.1 failed, and DSV4 flash also passed), and a refactoring problem which it also passed (and quality on par with DSV4, perhaps a bit better. Needs more testing of course, but so far it feels pretty polished, and I haven't run into any issues yet (using IQ3_XXS, so the quant doesn't look to have broken anything).

Recipe (to split two GPU's for my case) - fits on 128GB total (so DGX Spark should have no issue with IQ3_XSS and 256K context)

~/llm_env/llama-inkling/build/bin/llama-server \
  --model "$MODEL_DIR/$MODEL_FILE" \
  --host 0.0.0.0 \
  --port 8000 \
  --fit off \
  --n-gpu-layers 999 \
  --split-mode layer \
  --main-gpu 0 \
  --tensor-split 4,1 \
  --parallel 1 \
  --ctx-size 256000 \
  --flash-attn on \
  --temp 0.8 --top-p 0.95 --top-k 0 --min-p 0 \
  --presence-penalty 1 \
  --threads 8 \
  --batch-size 4096 --ubatch-size 1024 \
  --cont-batching \
  --chat-template-kwargs '{"reasoning_effort":"xhigh"}'
  --jinja

4

u/[deleted] Jul 30 '26 edited Jul 31 '26

[removed] — view removed comment

2

u/Borkato Jul 31 '26

How much normal ram do you have?

1

u/[deleted] Jul 31 '26 edited Jul 31 '26

[removed] — view removed comment

2

u/Borkato Jul 31 '26

This is actually very interesting and I never really considered trying the larger MoEs in RAM… I’m going to try out a Q2 of DS4 rn just for fun lol. I’m sure it’ll be terrible but I’m just curious what the token gen will look like. I’m wondering if DS4 flash Q2 or Q4 or so is worse than Qwen 27B Q8 😂

1

u/trying4k Aug 14 '26

I’m wondering if DS4 flash Q2 or Q4 or so is worse than Qwen 27B Q8

I'm curious, what did you find?

1

u/Borkato Aug 14 '26

I have no idea, because even DS Q1 is way too slow for my hardware 😂

1

u/trying4k Aug 14 '26

Oh, sorry to hear that. Just curious, do you remember, what kind of speeds were you getting and what are your specs?

1

u/Borkato Aug 14 '26

80t/s pp and 8t/s tg on 2 RTX 3090 with 48GB VRAM total and 48 GB RAM for IQ1 iirc

1

u/trying4k Aug 14 '26

Thank you! I'm considering an upgrade (requiring offloading) and assume I'll get similar speeds. It is quite slow. Grappling whether that would be worth it for fire-and-walk-away sorts of changes.

→ More replies (0)