Who llama-swap is for#
AI developers iterating across multiple local models
A developer might use a small coding model for inline suggestions, a larger chat model for reasoning tasks, and an embedding model for RAG. llama-swap lets all three share the same API endpoint and GPU, swapping between them on request rather than requiring manual process management.
Skip if:
If you only ever use one model at a time with no plans to switch, the overhead of a proxy layer adds no value. A direct connection to the inference server is simpler.
Teams building privacy-sensitive AI applications
Organizations that cannot send data to cloud inference APIs due to compliance requirements, confidentiality agreements, or internal policy use llama-swap to run model serving entirely on their own infrastructure. The OpenAI-compatible API means existing tooling connects without modification.
Skip if:
If your use case allows cloud inference and you need managed uptime, dedicated hosted services handle scaling and reliability in ways a self-hosted proxy does not.
Researchers comparing model variants on the same prompts
Research workflows often involve running the same prompt across several checkpoint variants or model families to compare outputs. llama-swap's profile switching and per-model aliases let researchers swap between variants via config, with model IDs remapped through the `aliases` field.
Skip if:
If your inference workload requires simultaneous parallel evaluation of many models, the default single-model-at-a-time mode will not fit. The `matrix` feature extends this, but it requires adequate hardware resources for all loaded models.
Home lab users consolidating AI tooling behind one endpoint
A home server running llama.cpp, stable-diffusion.cpp, and a Whisper-based transcription server can be unified behind a single API endpoint with llama-swap. TTL-based auto-unloading means GPU memory is freed when models are idle, which matters on machines with limited VRAM.
Skip if:
If your machine has enough GPU memory to run all models simultaneously without swapping, a simple reverse proxy over multiple model servers may be easier to maintain than a dedicated model manager.
The problem it solves#
Running more than one local AI model on a single machine requires managing separate server processes, separate ports, and separate startup sequences. llama.cpp, vllm, stable-diffusion.cpp, and similar inference engines each need their own process lifecycle. Switching models means stopping one server, waiting for it to shut down cleanly, and starting another. For developers iterating across a coding assistant, a chat model, and an embedding model in the same session, this back-and-forth is a constant interruption.
The deeper issue is memory. Most local inference servers load model weights into GPU or system memory when they start. Running two simultaneously means doubling the memory footprint, which most consumer hardware cannot sustain. The only practical option has been to pick one model per session and stick with it, which limits what local AI workflows can accomplish.
How it solves it#
On-demand model swapping
llama-swap reads the `model` field in every incoming API request and starts the matching inference server automatically if it is not already running. When a new model is requested, it stops the current server and starts the correct one. No client-side configuration change is needed: the same API call that works with a managed service works unchanged.
OpenAI and Anthropic API compatibility
Exposes the full OpenAI v1 endpoint set including chat completions, completions, embeddings, audio speech, audio transcriptions, image generation, and image editing. Also supports Anthropic's messages API and token-counting endpoint. Any client that speaks either API connects to llama-swap without modification.
Web UI with playground and live metrics
A built-in browser interface provides a model playground for testing prompts, real-time token metrics, request and response inspection, and live log streaming. Models can be loaded and unloaded manually from the UI. The Playground also includes a Docs tab backed by an MCP endpoint that answers configuration questions using the local knowledge base.
Multi-backend support
Works with llama.cpp, vllm, stable-diffusion.cpp, ComfyUI, audio.cpp, and any other OpenAI-compatible server. The `cmd` field in the config file defines how each model's server starts, which means any inference engine with a compatible API is supported without code changes to llama-swap itself.
Concurrent models via matrix DSL
A `matrix` configuration block lets you define which models can run simultaneously and how they share GPU resources. The default is one model at a time; the matrix adds a custom swap logic layer for workloads that need more than one model loaded concurrently.
TTL-based auto-unloading and request filters
Setting a `ttl` on any model causes llama-swap to unload its server after the specified idle period, freeing GPU memory automatically. The `stripParams`, `setParams`, and `setParamsByID` filter options rewrite parts of requests before forwarding them upstream, letting you normalize inference parameters across backends.
Strengths and trade-offs#
Strengths
- Single binary, zero external dependenciesThe entire proxy ships as a pre-compiled Go binary. There is no runtime to install, no language environment to configure, and no package manager to satisfy. Download the binary, write a YAML config file with a single required key, and run. This is the lowest-friction self-hosted infrastructure setup in the local LLM ecosystem.
- MIT license with no commercial restrictionsMIT means you can run llama-swap in commercial products, modify it, and redistribute it without paying licensing fees or agreeing to usage-based terms. Unlike some LLMOps tools that carry AGPL or proprietary licenses, llama-swap imposes no constraints on how you build on top of it.
- Drop-in API compatibility for any OpenAI clientBecause llama-swap presents a standard OpenAI/Anthropic API surface, every downstream tool (IDE plugins, chatbot UIs, code assistants, RAG frameworks) connects without modification. Switching the underlying model or inference engine is an infrastructure change only; client code does not need to change.
- Docker and Podman support with clean process lifecyclePython-based inference servers like vllm and tabbyAPI benefit from container-based process management. llama-swap's `cmdStop` flag sends the correct signal to stop Docker or Podman containers cleanly, which avoids orphaned processes and GPU memory leaks between model swaps.
Trade-offs
- -Manual configuration required for every modelEvery model must be declared in the YAML config file with the full command to start its server. There is no auto-discovery of models from a directory, no GUI to add models, and no connection to model registries. Teams with large model libraries need to maintain config file entries manually as they add or update models.
- -No built-in persistence or chat historyllama-swap is a proxy, not a frontend. It routes API requests and manages server processes but does not store conversation history, manage sessions, or provide a production chat interface. Users who need those features must add a separate layer (such as Open WebUI or Lobe Chat) on top of the proxy.
- -87 open issues; edge cases may have known bugsThe project has accumulated bug reports and feature requests faster than they are resolved. Active maintenance is evident (last push within 24 hours of this writing), but less-common backends and advanced configurations may have known issues that have not yet been addressed. Check open issues before depending on a specific backend.
llama-swap vs alternatives#
llama-swap vs Managed Cloud Inference APIs
Managed cloud inference services like Together AI, Fireworks AI, and Replicate expose OpenAI-compatible endpoints from their own GPU infrastructure. Enterprise options like Amazon Bedrock and Azure OpenAI do the same at larger scale with compliance certifications. llama-swap does neither: it runs on your own hardware and manages the inference servers you configure there.
| Feature | llama-swap | Managed Cloud APIs |
|---|---|---|
| License | MIT | Proprietary |
| Self-hosting | Required | Not available |
| Data leaves your network | No | Yes |
| Pricing | Free (hardware cost) | Per-token billing |
| Model selection | Any locally-runnable model | Provider catalog only |
| Uptime guarantee | None (self-managed) | Provider SLA |
| Offline operation | Yes | No |
llama-swap is the better choice when inference data cannot leave your network (compliance requirements, confidentiality agreements, or internal policy), when per-token costs at volume make cloud APIs expensive, or when you need to run models not available in any cloud provider's catalog. A research team running fine-tuned checkpoint variants or an organization that cannot send prompts to external servers will find llama-swap handles both use cases with no licensing restrictions.
Managed cloud APIs are the better choice when your team lacks GPU-equipped infrastructure, needs zero-maintenance uptime, or depends on frontier models that cannot run locally. Horizontal scaling, geographic distribution, and guaranteed latency SLAs are outside what llama-swap provides: it is a single-machine proxy with no built-in clustering or failover.
Install and self-host#
Install llama-swap with Homebrew on macOS or Linux, then point it at a config file.
```bash
brew tap mostlygeek/llama-swap
brew install llama-swap
llama-swap --config config.yaml --listen localhost:8080
```What it's built on#
- Languages
- GoTypeScript
- Frameworks
- Svelte
FAQ#
Does llama-swap require llama.cpp specifically?
No. llama-swap works with any inference server that exposes an OpenAI or Anthropic-compatible API, including vllm, stable-diffusion.cpp, ComfyUI, and audio.cpp. llama.cpp (via llama-server) is the most tested backend, but the README explicitly states that "any OpenAI compatible server would work." The architecture is backend-agnostic by design.
How does llama-swap handle GPU memory when swapping models?
llama-swap stops the current inference server before starting the next one, which releases GPU memory from the first model before the second loads. The ttl config setting automates this: after a model goes idle for a specified duration, llama-swap unloads it automatically. The matrix feature allows keeping multiple models loaded simultaneously, but that requires enough GPU memory for all of them.
Can I use llama-swap with Docker containers?
Yes. The cmd config field accepts any shell command, including docker run or podman run. The cmdStop field handles graceful shutdown by sending the correct signal to the container. The project also publishes its own Docker images via GitHub Container Registry, including a unified image with llama-server, stable-diffusion.cpp, and Whisper pre-built, available in CPU, CUDA, Vulkan, and Intel variants.
Is llama-swap suitable for production use?
llama-swap is MIT licensed and described in the README as used by thousands of people. It ships as a binary with no external runtime dependencies, which simplifies deployment. However, it does not provide managed uptime, horizontal scaling, or load balancing across multiple machines. Teams running it in production typically pair it with a reverse proxy like nginx (the README includes nginx config snippets for streaming endpoints) and a process supervisor.
Does llama-swap support the Anthropic API format?
Yes. llama-swap proxies the Anthropic v1/messages and v1/messages/count_tokens endpoints to backends that support the Anthropic API format. The upstream server must accept Anthropic-format requests; llama-swap is format-transparent on those routes and does not translate between OpenAI and Anthropic formats.
Similar open-source tools#
OpenCode
OpenCode is an open-source AI coding agent that assists developers in
magnitude
Local models for your agent, tuned for your hardware
sie
One self-hosted cluster for all the models your agents need
heretic
Decensor local LLMs automatically with directional ablation
marin
Open lab for training foundation models together
trueforge
Open-source agent harness for reliable, long-running LLM agents