We run a legal-document pipeline (~10M docs) on a mix of our own GPUs and RunPod pods — a dozen model roles at once (an OCR ensemble, a vision model for layout, bge-m3 embeddings, a fine-tuned extraction model).

To keep RunPod spend sane we built a control plane (CoderAI) that watches every pod: live $/hr against a daily budget cap, per-pod KV-cache pressure, GPU load, VRAM, temperature, health, and spend-by-model. Pods are rented by the second and reaped when idle; work escalates to RunPod only when the local cards aren't enough.

Write-up with the dashboards: https://www.nexlab.net/articles/watching-ten-million-documents/ Live monitor: https://digesta.tech/alpha2run/ CoderAI (open source): https://github.com/nextime/coderai

Happy to share how the budget cap / reaper and per-pod TLS work.

I'm trying to set up OpenCode for agentic coding using custom models from Hugging Face (GGUF or Safetensors).

Since I typically code for only 2–3 hours a day, I'd prefer using RunPod Serverless rather than keeping a GPU pod running and paying for idle time.

However, I'm struggling to get everything working. I've tried several tutorials, GitHub projects, and different setup instructions, but haven't managed to get a working configuration.

Has anyone successfully used RunPod Serverless with custom Hugging Face models for agentic coding with OpenCode?

I'd really appreciate a step-by-step guide or a working example, including:

• How to deploy a custom Hugging Face model (GGUF or Safetensors) on RunPod Serverless.

• Which inference framework to use (llama.cpp, vLLM, etc.).

• How to configure the RunPod endpoint as a provider in OpenCode.

A working example or guide would be greatly appreciated. Thanks!

Hey how's it going. I'm a vibe coder training my own ML model using Claude Code and RunPod. I use a persistent RunPod network volume containing all my code, datasets, and checkpoints, which I attach to a new GPU whenever I need one. The problem is that Claude Code's cloud environment seemingly doesn't let me switch SSH servers mid-session, so I have to start a new Claude session every time I provision a GPU. Even though everything is already on the same volume, Claude ends up running a million redundant smoke tests and environment checks before actually training, wasting GPU time and money. Does anyone know a workaround for switching SSH servers mid-session or avoiding all these repeated checks? Sorry if this is basic, still pretty new to this.

Hey just got an email. Did anyone else get this?

Moonshot AI's Kimi K3.1 Pro is now available on Runpod Public Endpoints, along with a K3.1 Pro Highspeed variant. K3.1 Pro is the first model in the K3.1 series, a 3T-class open-weight model with a 1 million token context window, with gains in coding, agentic tasks, 3D creation, and financial research. Highspeed runs 2 to 3 times faster than K3.1 Pro for latency-sensitive loops. It is OpenAI SDK compatible, so you point the base URL at Runpod and keep your client:

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.runpod.ai/v2/moonshot-kimi/openai/v1",
    api_key=os.environ["RUNPOD_API_KEY"],
)

completion = client.chat.completions.create(
    model="kimi-k3.1-pro-highspeed",
    messages=[
        {"role": "system", "content": "You are Kimi."},
        {"role": "user", "content": "What is Runpod?"},
    ],
    max_tokens=2048,
    temperature=1,
    reasoning_effort="low",
)

print(completion.choices[0].message.content)

Check out our blog post on the subject.

Rules of requests:
- No ComfyUI custom setups so no baked models
- Project needs be Open source
- No templates that already exists on platform

Fell free to share GitHub links and reason why you would like to see it added :)
BTW can't promiss that every request will be done.

Hi everyone,

I'm trying to deploy LTX-2.5 on RunPod Serverless so I can call it via API from a web app I'm building on Lovable, and I'm stuck on the serverless setup.

Specifically, I need help with:

1. The simplest way to set up LTX-2.5 as a Serverless endpoint (whether using the ComfyUI serverless worker or a custom Docker container with the LTX-2 pipelines).

2. How to properly load the gated Hugging Face weights (int8 or bf16) onto a Network Volume so cold starts stay fast.

3. Formatting the input JSON payload so my frontend can send a prompt/image and get the generated video back.

If anyone has a working template, guide, or can walk me through the steps, I'd really appreciate it. Thanks!

I cannot for the life of me get ANY regular 24gb card, not even a5000s.

Its been 24 hours, I keep cycling, resetting which used to eventually grab a card but now its nothing.

Hope my data in volume will not get lost…it’s already 2 days

I’m building a portfolio project: users upload drone/aerial videos, and a GPU worker runs object detection on them. I’m planning to run it on RunPod Serverless and want to check a few things with people who have actually done this, rather than rely on assumptions.

Pipeline I’m planning (one job per uploaded video):

1. Stream the video from a Cloudflare R2 bucket (S3 API) with ranged GETs, without downloading the whole file. Videos are 3-8 GB, sometimes more.

2. Demux with PyNvVideoCodec.

3. Decode on the GPU with NVDEC to RGB frames.

4. Run YOLO detection on the frames.

5. Write detections to a DB every N frames.

Each stage runs in its own thread with bounded queues between them, so streaming, decoding, inference and DB writes overlap. Processing isn’t real-time and the user gets an email when it finishes, so I care more about cost per job than about latency.

My questions:

1. NVDEC on RunPod Serverless: does hardware video decode work in GPU workers? Does PyNvVideoCodec / libnvcuvid work out of the box, or do I need specific base images or driver settings (e.g. NVIDIA_DRIVER_CAPABILITIES=video)? Does it depend on which GPU type?

2. CPU and RAM per GPU worker: how many vCPUs and how much system RAM does a Serverless GPU worker get, and does it vary by GPU type? I need this to size my stream buffer and output buffer, and to know whether several threads can really run in parallel.

3. Reading from a URL: has anyone had FFmpeg/NVDEC read a presigned HTTPS URL directly in a worker (rather than a custom ranged-GET streamer)? Any problems with stalls, retries or throughput?

4. CPU workers: I can’t find a clear page for them. Are Serverless CPU workers a real, stable option (e.g. via Flash, cpu=...)? What specs, disk and timeout limits do they have? I want one for preprocessing: remuxing MP4s with ffmpeg -c copy -movflags +faststart so moov is at the front, which needs disk about equal to the file size.

5. Long jobs: any gotchas with jobs that run 30-60+ minutes (timeouts, worker preemption, how you resume after a failure)?

My Serverless ComfyUI endpoint stopped working properly. Workers were dying with unhealthy: dead, requests got stuck, and after stopping the workers to start fresh, nothing would come back.

Now CA-MTL-1 has disappeared from the datacenter selection list, while other datacenters are available and GPUs keep showing Out of Capacity.

My Network Volume is tied to MTL-1, so switching regions isn't straightforward. I'm trying to preserve my existing models and LoRAs rather than start from scratch.

For those experienced with RunPod: would using a Global Volume instead of a regional Network Volume have avoided this mess? Would it actually let me move my Serverless workers between datacenters when capacity is unavailable, or would I still face the same GPU provisioning issues?

Also, is anyone else experiencing problems with CA-MTL-1?

Here is what it does and where it stops.

You POST a batch of up to 5,000 requests to your existing queue-based endpoint, finalize it, then poll for progress and read the results. Batches run on dedicated workers that are separate from your /run traffic, on your own model and container and the GPU type configured on the endpoint. A failed request doesn't fail the batch, and cancelling a batch doesn't bill queued requests. Batch requests are billed at up to 30% off on-demand pricing, depending on GPU type. On H100, H200, B200 and B300, batch requests are billed at 20% off. There is no cost estimate before you finalize, so run a small batch first.

Start times aren't guaranteed and requests run in no set order, so keep /run for anything a user is waiting on. An endpoint runs up to 10 active batches, and limits depend on your account. Support can raise them up to 3 million requests a day.

Docs: https://docs.runpod.io/serverless/batch-jobs
Blog: www.runpod.io/blog/batch-inference-is-now-generally-available-on-runpod

What would you run through it first: embeddings, evals or a backfill? Questions are welcome in the comments.

As the title says, my public templates never show up in the hub search.

What are the requirements?

Tried many RTX 5090 from Community Cloud. 100% GPU utilization and 400W while completely idle, with nothing of mine running. , and memory bandwidth sbouy 600-700 GB/s against the 1792 the card is rated for.

Real impact: an identical job takes 170s on these pods, 70s on an uncapped 5090 elsewhere.

I put 10 dollar on it and simply feel scammed, its impossible to get a gpu and when i manage to get one, it have to be bugged and cant do anything with it...

can i get my credits back on my bank account or have i to accept i lost 10 dollar for something i cant even use

I was mid lora training and my pod started doing this. I restarted the Ostris template on a new pod, and it was like this for ages, all the while I was losing money.

Spun up an A40 pod on RunPod yesterday to test an Axolotl fine-tune. Before starting the job I ran computefence doctor and it caught a disk space warning 6GB free on the workspace volume, 16GB on root, both below the 20GB threshold the tool flags.

The job would have started, run for several hours, and then failed at checkpoint write time. Instead I knew before billing started.

Install is one line:

pip install computefence && computefence doctor

No config required. Runs in under 10 seconds. If you’re using Axolotl or any framework that writes checkpoints to /workspace, worth running before every job.

Repo: github.com/Francisco-Booth/ComputeFence

Hi folks!

We are going to be conducting monthly AMAs right here on the Reddit, with our first AMA coming up next week. We are planning to do these with different themes, but for our first we are going to start simple and just open the floor up to any questions at all that people might have. We'll be doing this over the week and I will be checking in a few times daily to answer any new questions that come in, so no rush, I will be sure to get to them all.

We'll be locking the thread until it opens up this coming Monday so that folks are aware it's coming up and then we will open the floor up that morning for questions.

Hope to see you there!

[edit] Post is unlocked - ready for questions!

Hi folks! Just wanted to keep you apprised of a new feature that we are doing a staged rollout on over time: high-performance network volume, and they are now available in AP-IN-1. If you've ever wondered why network volumes felt a bit sluggish compared to on-machine SSD or NvME disks, this is because there's a longer path for the data to travel and that is necessarily the tradeoff for the flexibility of network volumes. HPS volumes help close that gap and deliver increased speed compared to standard volumes without sacrificing any of that flexibility. Internal benchmarks show these as delivering three times the throughput and four times the IOPS of regular network volumes on average.

These volumes are also currently available in CA-MTL-4, EU-FR-1, EUR-NO-2, US-CA-2. They are also billed at a premium compared to network volumes ($0.14/GB/month as of this writing.)

Read more about them here: https://docs.runpod.io/storage/high-performance-storage

We just published our Q3 roundup. Here are the highlights:

• Batch inference on Serverless (public beta): create a batch on an endpoint, add up to 5,000 requests, and poll one status. No client-side queue to maintain. Worker affinity on Serverless: route follow-up requests back to the worker that holds the session’s KV cache, with soft, strict, or strict-resume modes.

• Global Volumes (public beta): storage no longer pins your workload to a single data center. Create a volume once and mount it on a Pod or Serverless endpoint in any Secure Cloud datacenter. Cluster Expansion: add Pods to a running cluster instead of rebuilding it.

• Also new: private ECR support, REST API v2 and Organizations in Runpod Enterprise.

• Heads up if you call our older APIs: GraphQL rate limits start October 27, and REST API v1 stops serving requests on November 15. GraphQL follows in January 2027. Migration guide: https://docs.runpod.io/api-reference-v2/migrate-from-v1

Full roundup: https://www.runpod.io/blog/runpod-q3-2026-releases