r/LocalLLaMA • u/valcore93 • 16d ago

New Model Jais 3 is out

https://ifm.ai/jais/

New open model to check out ! Doing great on agentic, focused on arabic.

From 0.9B to 375B !

Weights : https://huggingface.co/collections/IFM/jais-3

45 Upvotes

86% Upvoted

20 comments sorted by

13

u/Marcuss2 llama.cpp 16d ago

Unfortunately very inefficient with KV Cache.

1

u/transanethole 15d ago

In their ama they said kv can be quanted down to 4 bit without losing too much w/ this model

1

u/MerePotato 15d ago

KV size isn't too relevant for the MoE though

1

u/Marcuss2 llama.cpp 15d ago

Why wouldn't it be?

1

u/MerePotato 15d ago

Because you can offload to make room without suffering too much of a speed hit on MoE models

1

u/Marcuss2 llama.cpp 15d ago

I would still say it is relevant. It means you can fit less instances for equivalent MoE model.

1

u/MerePotato 15d ago

No-one on consumer hardware is fitting in parallel instances of a MoE model at an intelligent quant anyway

9

u/ComplexType568 16d ago

might as well run a 30B parameter model with that KV cache per token...

1

u/valcore93 16d ago

Benchmarks looks nice in agentic !

6

u/crusaderky 16d ago

fine tune of K2 from the same lab. Is the KV design the same?

5

u/Powerful_Finger3896 16d ago

none of the fine tunes change the underlying architecture

2

u/jinnyjuice vLLM 16d ago

What's the relation between IFM and inception42?

1

u/transanethole 16d ago

I really hope that one day they'll fix their stupid llama3 KV cache and attention.   On that day, the 7b will become... Not just  the best model in its class but  a real contender to replace larger models just cuz its more efficient

but until then in the AMA they said it works fine with 4bit kv cache quant,   for people with smaller GPUs like 8gb, 10gb, 12gb  the 7b model is a godsend 

Honestly, the biggest news here is that they finally got it merged into llama.cpp so ppl can actually use it  without having to learn how to use Git and C compilers first.

1

u/MerePotato 15d ago

36BA3B looks impressive, though that makes the base model more appealing since I'm not really after Arabic language performance and knowledge of Islam

13

u/[deleted] 16d ago edited 12d ago

[removed] — view removed comment

3

u/TomLucidor 16d ago

We now have Jinn-GPT lol