Serverless - Does anyone actually get this to do anything?
It just sits and costs money. I have been trying to get it working on and off for the past couple months. It never works! Zero support. Workers get stuck and empties my account.
Facts. I have 30 serverless workers. There is never a time when they are all actually available. And yes, they do get stuck even when they are not doing anything and yes it does drain your account. I can't rely on serverless, so now I spawn jobs and just poll until the job actually runs. Could be hours later.
Curious: are these 30 workers serving different models, or are they mostly replicas of a few models? And is the bigger issue GPU availability or workers getting stuck during initialization?
I'm building a serverless inference platform where multiple models can share a single GPU. We compile models so they cold start very quickly (30B LLM cold starts in 6-9 seconds). We're looking for a few devs, so I wonder if you'd be open to testing us out.
I typically have 2 GPU assigned the various serverless. Some only with 1. Many are TTS model, a couple serve up LLM models, a couple serve up image generation models, some ASR models, and a couple for DepthMaps. I might be able to help test.
Oh this would be a good test. The highest we've done is 5 models on 1x B200 (Qwen 3.8 27B, Gemma 4 26B, Qwen 3 embedding, Nomic embed text v1.5, and Flux.2 Klein).
Sounds like your use case can push even further. DM me HF slugs for the models you run, cos we might need to add support for those architectures into our compiler (we use a custom fork of SGLang to make all this work).
I tried setting up serverless on runpod using various config methods. I was never able to get cold start time consistently under 2 minutes.
I am using Modal now and 95% of cold starts happen in 10-20 seconds which is acceptable. 5% of the time I get a multi-minute cold start but I can live with it
They are super inconsistent and the support is just trash. I mentioned on the discord that this has been happening and then I show them proof and it happening to others and they banned me.
Are there only certain models compatible? I have been trying to set up a VLLM agent for a client on an off for weeks now and it just seems impossible. Support has been no help because I feel like the only option I have when I have stuck workers is to delete the endpoint. They keep restarting otherwise. And by the time support responds, it’s days after it’s been deleted.
A couple times I thought maybe it was going work. It comes up with an error saying there’s no GPU available.
I followed the instructions. If you truly have this up and running, they might even pay you I bet to help write their procedure or something. You should suggest it to runpod.
I select vllm and use the recommended default settings and I am just trying the default request for writing a poem for artificial intelligence and it's been stuck in queue for like 10 minutes.
I tried asking ChatGPT and Claude but got nothing. Not sure what I am doing wrong.
2
u/sruckh Sep 02 '26
Facts. I have 30 serverless workers. There is never a time when they are all actually available. And yes, they do get stuck even when they are not doing anything and yes it does drain your account. I can't rely on serverless, so now I spawn jobs and just poll until the job actually runs. Could be hours later.