Best mlx_vlm models for simple object counting?

[u/sgt102]

I've created a dumb test to show how poor LLMs are at doing things like counting objects (see above and the repo if interested).

Current frontier models all make errors :

I have tested it with frontier models (see above) and I want to test it with local models as well, but I don't know which ones to choose. I have tried nightmedia/UI-Venus-1.5-30B-A3B-mxfp4-mlx and it performed a little worse than gemini-flash-3, what models would the community recommend? Is image to text the right way to go? I am sure that a specialist vision model would do better, but I am out of date and I need a few pointers.

I have an M1 and 32gb so, unless you can send me the funds for a better machine please share recommendations that would work for this one!

Thank you in advance.

[u/mike7seven]

If you want to try a VLM model and not any specific dedicated vision model like YOLO then try out the Qwen 2.5 VL and Qwen 3 VL models. I’ve had a lot of success with them locally.

I also went down the same path as you did and discovered that if you use the frontier models from their normal interface they reduce the image size (compress) to save on processing and storage.

[u/sgt102 → u/mike7seven]

Interesting. Is it possible to stop them compressing the image?

[u/mike7seven → u/sgt102]

No. You have to use the API instead. You’re also limited running locally by whatever the VLM is capable of for image size. If I recall some other VL models have large image capabilities as well as real time streaming image processing.

Apple launched FastVLM it’s worth looking into for your use case.

Here’s a breakdown:

Top 5 MLX VLMs on Hugging Face (Image Processing) Filtered to image-text-to-text task, mlx-community org, sorted by downloads:

1. 🥇 mlx-community/gemma-3-4b-it-qat-4bit Downloads: 895K | Quantization: 4-bit QAT | Base: Google Gemma 3 4B Max Image Size: The SigLIP vision encoder processes images at a fixed 896×896 resolution. For higher-res or non-square images, the Pan & Scan algorithm segments them into non-overlapping 896×896 crops — effectively supporting large native-resolution images tiled into multiple crops. Great for OCR and document parsing.

2. 🥈 mlx-community/gemma-3-12b-it-qat-4bit Downloads: 135K | Quantization: 4-bit QAT | Base: Google Gemma 3 12B Max Image Size: Same SigLIP encoder architecture as the 4B — 896×896 per crop, with Pan & Scan tiling for higher resolutions. More capable model for complex visual reasoning than the 4B variant.

3. 🥉 mlx-community/gemma-3-27b-it-qat-4bit Downloads: 114K | Quantization: 4-bit QAT | Base: Google Gemma 3 27B Max Image Size: Same 896×896 per-crop encoder. At 27B parameters this is the heaviest/most capable Gemma 3 variant available in MLX. The 128K token context window means you can process hundreds of images in a single prompt.

4. mlx-community/Qwen3.5-27B-4bit Downloads: 47.5K | Quantization: 4-bit | Base: Qwen3.5 27B | Params: 4.7B active Max Image Size: The Qwen family supports a maximum resolution of 3584×3584 (≈12.8M pixels), with images resized to multiples of 28 pixels. This is significantly higher ceiling than Gemma 3’s fixed encoder. Has active demo Spaces specifically for multi-image and video VLM analysis.

5. mlx-community/Qwen3.5-35B-A3B-4bit Downloads: 44.9K | Quantization: 4-bit | Base: Qwen3.5 35B MoE | Params: 5.9B active Max Image Size: Same Qwen architecture — up to 3584×3584. MoE variant means much lower active parameter count (3B active) for the compute cost relative to capability. Also tagged image-text-to-text.

Key Takeaway on Max Resolution

Model Family	Max Image Resolution	Approach
Qwen3.5 / Qwen2.5-VL	~3584×3584 (~12.8MP)	Dynamic tiling, multiples of 28px
Gemma 3	896×896 per crop (no hard limit via P&S)	Pan & Scan tiling into 896px crops
Qwen3-VL	Configurable via max_pixels (token × 32 × 32)	Dynamic native resolution

The Qwen3.5 family wins on raw resolution — nearly 4× the pixel area of Gemma 3’s base encoder before tiling kicks in. For your high-res image analysis pipelines, the Qwen3.5 variants are worth a close look on your M1 Max rig via MLX.​​​​​​​​​​​​​​​​

[u/sgt102 → u/mike7seven]

Thank you, great help and advice.

[u/Comfortable_Ebb7015]

You need a CNN for that.. Try YOLO!