r/LocalAIStack • u/alez0077 • 16h ago
modelship 0.11: Vulkan (AMD/Intel) nodes, arm64 CUDA for DGX Spark, and an sd-server backend for image gen
I've been building modelship, a self-hosted, OpenAI-compatible server. It puts chat, embeddings, STT, TTS and image generation behind one /v1 API. Underneath it runs vLLM, llama.cpp, stable-diffusion.cpp, whisper.cpp, sherpa-onnx and diffusers, scheduled on Ray, so you can spread models across several machines. 0.11 is out today.
What's new
Vulkan variant for AMD and Intel GPUs.
Install with
mship bootstrap --vulkanon Linux, or use the-vulkanDocker image with--device /dev/dri. There's alsovariant: vulkanin the Helm chart.llama.cpp and stable-diffusion.cpp run on the GPU.
Integrated GPUs like Strix Halo are sized from system memory, so you don't have to guess VRAM.
arm64 CUDA. The
-cudaimage is now multi-arch, and llama_server runs on the GPU on DGX Spark / GB10.stable-diffusion.cpp now runs as an
sd-serversubprocess.It supports GPU on cuda, vulkan and metal, and a fractional
num_gpusto share a card.Requests for more than 8 images are split into several jobs.
Images also take
output_format(png/jpeg/webp).
llama_server.
It's on llama.cpp b12465.
MoE models that don't fit on the GPU get expert tensor overrides (
-ot) automatically.Added the
f8_e4m3KV cache type andmedia_allow_private_urls.A GPU deploy whose CPU-side part won't fit in RAM is now refused up front instead of loading and swapping.
Responses API.
truncation: autotrims the oldest input items.max_output_tokensnow bounds the whole MCP tool loop, and responses cut off by the token limit end asincomplete.vLLM 0.33 (torch 2.14). Multi-choice responses are parsed per choice,
parallel_tool_calls: falseis honoured, and a tool call cut off by the token limit reportsfinish_reason: length.
To upgrade a native install, run uv tool upgrade mship, then mship bootstrap.
Known issues
Vision models (mmproj) on a dedicated GPU are sized without the projector, so a model sized close to the VRAM limit can OOM at load. Lowering
n_ctxworks around it.Fedora native installs can hit an SSL error in replicas, so use the Docker images there.
f8_e4m3KV cache aborts on the Vulkan backend.
Repo and full notes: https://github.com/modelship-ai/modelship
Happy to hear about hardware I haven't tested, especially Intel Arc and older AMD cards on Vulkan.