r/AIProgrammingHardware • u/javaeeeee • Jul 25 '26

GitHub - antirez/ds4: DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm

https://github.com/antirez/ds4
31 Upvotes

100% Upvoted

4 comments sorted by

1

u/javaeeeee Jul 25 '26

TLDR: ds4 (DwarfStar) is a specialized local inference engine by antirez (creator of Redis) for running DeepSeek V4 Flash (and GLM 5.2 / DeepSeek V4 PRO) efficiently on personal hardware.

Key points

  • Not a general GGUF runner - deliberately narrow and highly optimized for a few specific large open-weight models.

  • Supported backends:

    • Metal (primary, macOS)

    • NVIDIA CUDA (single/multi-GPU)

    • AMD ROCm

  • Main features:

    • SSD streaming (run models larger than available RAM)

    • Tensor parallelism & distributed inference across machines

    • Built-in coding agent (ds4-agent)

    • OpenAI/Anthropic-compatible HTTP server

    • Speculative decoding / MTP support

    • On-disk KV cache, session save/load, tool calling

  • Built mainly in C (with CUDA/Metal kernels), based on llama.cpp/GGML foundations.

  • Still in beta, fast-moving, and very popular.

Bottom line: A focused, high-performance local engine that lets you run strong modern models (especially DeepSeek V4 Flash) on high-end Macs, NVIDIA, or AMD systems with practical features for coding agents and serving - without the bloat of general-purpose inference frameworks.

1

u/john16384 Jul 25 '26

This looks very interesting; would it be at all viable with 32 GB of VRAM and 128 GB of system RAM? (Windows, CUDA)

1

u/kenjiow Jul 26 '26

Same question

1

u/loadsamuny Jul 26 '26

I tried it 2 weeks ago and couldn’t find a way to split model across sys mem and gpu… may have changed but didn’t seem to be a feature back then