Who DwarfStar (ds4) is for#
Individual Developer/Researcher
Run powerful DeepSeek V4 Flash or GLM 5.2 models locally on high-end consumer hardware like MacBooks (M-series) or Strix Halo systems, even with SSD streaming for memory-intensive tasks.
Small to Medium Business (SMB)
Deploy a multi-user LLM server using existing or older NVIDIA CUDA cards (e.g., L40S) with ds4-server's micro-batching and multi-GPU support, providing cost-effective internal AI capabilities.
AI Enthusiast with Multiple Machines
Combine the RAM and processing power of multiple systems (e.g., two MacBooks via Thunderbolt) using pipeline or tensor parallelism to run DeepSeek V4 PRO or other large models that would not fit on a single host.
AI Agent Development
Utilize the native coding agent for low-latency, integrated development and testing of AI agents, leveraging DeepSeek V4 Flash/PRO's tool calling capabilities with persistent sessions.
The problem it solves#
Running large, capable open-weight language models like DeepSeek V4 Flash/PRO or GLM 5.2 efficiently on consumer-grade hardware or older server GPUs often presents challenges due to memory constraints, performance bottlenecks, and the complexity of setting up specialized inference environments. Users need a highly optimized solution that can leverage available hardware, scale across multiple machines, and provide a seamless experience for local development and deployment.
How it solves it#
Optimized Model Support
Specifically tuned for DeepSeek V4 Flash, DeepSeek V4 PRO, and GLM 5.2 GGUF models, ensuring high-quality 2-bit quantizations and efficient execution.
Multi-Platform Hardware Acceleration
Supports Metal on macOS (M-series Macs), NVIDIA CUDA (including multi-GPU and DGX Spark), and ROCm on AMD Strix Halo systems for broad compatibility.
SSD Streaming for Large Models
Enables running models larger than available RAM by streaming routed MoE experts from fast local SSDs, making long contexts practical on machines with less memory.
Distributed Inference Capabilities
Offers both pipeline parallelism to sum RAM across multiple systems for larger models and tensor parallelism to split per-layer work between two Macs for reduced latency.
Native Coding Agent
Features an integrated coding agent with low latency, native tool calling, and persistent KV cache sessions, designed vertically for DeepSeek V4 Flash and PRO.
OpenAI/Anthropic-Compatible Server
Provides a local HTTP server with OpenAI/Anthropic API compatibility, supporting batched sessions and disk KV caching for multi-user LLM serving.
Strengths and trade-offs#
Strengths
- High Performance and EfficiencyAchieves impressive prefill and generation speeds on supported hardware, leveraging specialized optimizations and aggressive quantization for efficient inference.
- Memory OptimizationEffectively handles models larger than RAM through SSD streaming and compressed KV caches, making powerful LLMs accessible on consumer machines.
- Scalability for Larger ModelsSupports distributed inference via pipeline and tensor parallelism, allowing users to combine resources from multiple machines to run very large models.
- Multi-User ServingThe ds4-server with micro-batching and multi-GPU support can transform older CUDA cards into capable multi-user LLM servers, extending hardware lifespan.
- Specialized and IntegratedBeing narrowly focused on specific models allows for deep optimization and tight integration of all components, from model loading to the coding agent and HTTP server.
Trade-offs
- -Limited Model SupportDwarfStar is not a general GGUF loader and only works with specific DeepSeek V4 and GLM 5.2 GGUFs, requiring specific tensor layouts and quantization mixes.
- -Beta Quality SoftwareThe software is currently fast changing and considered beta quality, meaning instabilities are possible despite pre-release QA runs.
- -Distributed Generation LatencyWhile distributed inference helps fit larger models and speeds up long prefills, generation is strictly autoregressive and can be slower than a single local process due to cross-machine activation hops.
- -AI-Assisted DevelopmentThe project openly states strong assistance from AI models (GPT 5.5, 5.6, Claude Fable) in its development, which might be a concern for users preferring purely human-written code.
DwarfStar (ds4) vs alternatives#
DwarfStar acknowledges its foundational debt to llama.cpp and GGML, stating that it would not exist without the path opened by the llama.cpp project, including its kernels, quantization formats, GGUF ecosystem, and engineering knowledge. Some source-level pieces are retained or adapted from llama.cpp under the MIT license.
In the context of multi-user LLM serving, DwarfStar highlights its ability to turn servers with older CUDA cards (e.g., Ada Lovelace architecture) into multi-user LLM servers, noting that these cards are "no longer supported for new models by vLLM." This suggests DwarfStar offers an alternative for leveraging hardware that might be phased out by other inference solutions like vLLM for newer models.
Install and self-host#
make # macOS Metal
make cuda-spark # Linux CUDA, DGX Spark / GB10
make cuda-generic # Linux CUDA, other local CUDA GPUs
make strix-halo # Linux ROCm, AMD Strix Halo
make cpu # CPU-only diagnostics buildWhat it's built on#
- Languages
- CC++Objective-CPython
FAQ#
What models does DwarfStar support?
DwarfStar is optimized for DeepSeek V4 Flash, DeepSeek V4 PRO, and GLM 5.2 GGUF models. It is not a general GGUF loader and only works with specific, tested GGUF files.
Can I run models larger than my computer's RAM?
Yes, DwarfStar supports an SSD streaming capacity mode on Metal and for GLM 5.2 on ROCm. This mode keeps non-routed weights resident while streaming routed MoE experts from an in-memory cache loaded from the GGUF file on cache misses.
How does distributed inference work?
DwarfStar supports pipeline parallelism to split transformer layers across multiple machines to run larger models, and tensor parallelism to split per-layer work between two GPUs for reduced latency on the same token.
Is DSpark speculative decoding supported?
Yes, DSpark speculative decoding is supported for DeepSeek V4 Flash models. It uses an auxiliary draft model to propose future tokens, which are then verified by the main model for faster generation, especially for predictable continuations like code.
What is the current status of the DwarfStar software?
The software is currently in beta quality and is undergoing rapid changes. While extensive QA runs are performed before each release, instabilities are definitely possible.
Similar open-source tools#
Embabel
Agentic AI framework for the JVM
Reasonix
Open-source AI coding agent you can leave running locally.
AirLLM
Run huge LLMs on low-end GPUs with minimal VRAM
LMCache
Accelerate AI applications with caching technology
headroom
Compress LLM context before it reaches the model
jcode
Next-gen coding agent harness for efficient workflows