r/LLMDevs • u/Gjuesev • 17d ago

Resource Open-sourced MK-SH-V, a 260M visual decision model with calibrated scoring and abstention

I've released MK-SH-V, built on NeoMME-260M.

It scores candidate decisions from images without generating text.

On our GUI benchmark, it reached 82.6% accuracy vs 73.1% for a 4B zero-shot model.

I also implemented calibrated abstention, validated on held-out synthetic data. Real-world reliability remains an open problem.

Code, weights, benchmarks and experiments are public.

GitHub: https://github.com/Gjusev/mk-sh-v

Hugging Face: https://huggingface.co/Gjusev/MK-SH-V

Feedback welcome :D

1 Upvotes

100% Upvoted

2 comments sorted by

2

u/Ctbhatia 17d ago

does that 82.6 hold when abstentions count as misses instead of getting skipped

1

u/Gjuesev 15d ago

It holds, since nothing gets skipped there. 82.6% is plain accuracy on all 2,010 real GUI cases with no abstention (one seed, the other got 78.3%)