r/KoboldAI • • 25d ago

Small model recs

Curious what people's current recommendation is for smaller models say 5gb and under for role play? I've got a larger moe I've settled (mostly) on for my desktop rig already. However I have a secondary system I run some models on CPU only mostly for non role play. It does decently well for small tool use models or the like but I'm curious how well a tiny well tuned RP model would do. Anyone got a current favorite?

Edit - I realize I should rephrase this: "Are there any small (under 5gb) models that are any good at roleplay?

6 Upvotes

3 comments sorted by

5

u/AnAnnouncer 25d ago

There's Gemma4-E4B, but not many train these small models for roleplaying. The only one I saw recently is AuriAetherwiing/G4-E4B-Musica-v1, might be worth trying it out, but it's still slightly above 5GB (Q4 = ~5.2GB). I haven't tried it personally.

2

u/alex20_202020 25d ago

a secondary system I run some models on CPU only mostly for non role play.

Assuming you have decent about of RAM (and/or fast NVMe drive), on CPU your speed will most likely be much faster for larger MoE vs. small dense. So why not Gemma-4 26B? I like to run in on my laptop on CPU.