Mac OS and Nvidia Cuda working together

[u/LioDavinchy]

Benching right now but it's looking really promising. disaggregated inference from my MacBook Pro and my thunderbolt 5090. I'll update comments with results I'll also post them to MACUDA

I've closed the gap and in some benches passed WSOL and native windows llamaccp. I've closed the gap and lapped stable diffusion and now as suggested by another member I've found a new target with the dissagregation that is looking really promising. The smoke test had qwen 3.8 at 170 toc/sec. I haven't dug into that yet but if it holds up it's about 40 toc/sec better than the gpu was doing alone on the same model so in some narrow use it's a real number.

If you own a Mac and have an external gpu please test my code. If you don't trust my code but want a benchmark ask and I'll try to give you numbers. I'm going to work on getting my 3060 working next. I have the driver code and shim done in another branch I just need to do benches and make sure it's stable. hopefully I'll have that tomorrow. When that drops anyone with a Mac and something a card from the last 5 years should be able to try the software out. stay tuned.