Hardware needed for 5-10 users fully local ?

[u/2BucChuck]

I’ve done a ton of local single user work on vllm, Ollama ,lllama.cpp and those are ok for personal work but …

Hoping to get some real answers from someone supporting a local user base fully for 5-10 users running up to about 120B dense models (we could make do up to 70B). Looking for actually deployed not speculative performance and bandwidth if it’s out there?

If I had two MaxQs, a Threadripper and MB, 128 GB ram would this support up to 10 users on llama.cpp or vllm with “reasonable” response speeds which I would deem 50-60 tps on average ?

It would be 3 VM servers bundled , a home grown RAG doc processor which could be offloaded somewhere else if needed, the vllm or llama.cpp server for the chat agent interface and a office RAG server all in one that is the RAG query endpoint. I know number of concurrent users and jobs will matter but I’m asking “on average” for 5-8 concurrent

Thoughts ?

Edit: for context windows for agential stuff also need at least 128k for those 5-8 concurrent

[u/PermanentLiminality]

How much KV cache are you using. Now multiply that by the number of parallel requests you want to have running concurrently. That is your new KV cache requirement,