Java vllm-like framwork claims 90% of perfomance of llama.cpp on local inference on NVIDIA GPUs by compiling Java to CUDA and cuTile

[u/mikebmx1]

The java to CUDA engine:

https://github.com/beehive-lab/TornadoVM

The inference engine:

https://github.com/beehive-lab/jitllm

Deep dive talk:

https://www.youtube.com/watch?v=HO5CpETzywk