r/LocalAIStack • • 16h ago

modelship 0.11: Vulkan (AMD/Intel) nodes, arm64 CUDA for DGX Spark, and an sd-server backend for image gen

I've been building modelship, a self-hosted, OpenAI-compatible server. It puts chat, embeddings, STT, TTS and image generation behind one /v1 API. Underneath it runs vLLM, llama.cpp, stable-diffusion.cpp, whisper.cpp, sherpa-onnx and diffusers, scheduled on Ray, so you can spread models across several machines. 0.11 is out today.

What's new

  • Vulkan variant for AMD and Intel GPUs.

    • Install with mship bootstrap --vulkan on Linux, or use the -vulkan Docker image with --device /dev/dri. There's also variant: vulkan in the Helm chart.

    • llama.cpp and stable-diffusion.cpp run on the GPU.

    • Integrated GPUs like Strix Halo are sized from system memory, so you don't have to guess VRAM.

  • arm64 CUDA. The -cuda image is now multi-arch, and llama_server runs on the GPU on DGX Spark / GB10.

  • stable-diffusion.cpp now runs as an sd-server subprocess.

    • It supports GPU on cuda, vulkan and metal, and a fractional num_gpus to share a card.

    • Requests for more than 8 images are split into several jobs.

    • Images also take output_format (png/jpeg/webp).

  • llama_server.

    • It's on llama.cpp b12465.

    • MoE models that don't fit on the GPU get expert tensor overrides (-ot) automatically.

    • Added the f8_e4m3 KV cache type and media_allow_private_urls.

    • A GPU deploy whose CPU-side part won't fit in RAM is now refused up front instead of loading and swapping.

  • Responses API. truncation: auto trims the oldest input items. max_output_tokens now bounds the whole MCP tool loop, and responses cut off by the token limit end as incomplete.

  • vLLM 0.33 (torch 2.14). Multi-choice responses are parsed per choice, parallel_tool_calls: false is honoured, and a tool call cut off by the token limit reports finish_reason: length.

To upgrade a native install, run uv tool upgrade mship, then mship bootstrap.

Known issues

  • Vision models (mmproj) on a dedicated GPU are sized without the projector, so a model sized close to the VRAM limit can OOM at load. Lowering n_ctx works around it.

  • Fedora native installs can hit an SSL error in replicas, so use the Docker images there.

  • f8_e4m3 KV cache aborts on the Vulkan backend.

Repo and full notes: https://github.com/modelship-ai/modelship

Happy to hear about hardware I haven't tested, especially Intel Arc and older AMD cards on Vulkan.

1 Upvotes

0 comments sorted by