Who Switchyard is for#
AI engineers routing Claude Code or Codex to open-source models
Switchyard's launcher mode wraps Claude Code or Codex CLI invocations, translating their API calls to an Ollama, vLLM, or NVIDIA NIM backend. The agent keeps its native API; only the routing config changes. This lets a team evaluate open-source model quality on real coding tasks without modifying the agent's configuration.
Skip if:
Skip if the open-source models available to you cannot match the quality bar your workflow requires, or if Switchyard's pre-alpha stability is unacceptable for your team's development environment.
Research teams benchmarking multiple LLMs on identical traffic
Switchyard's random routing splits traffic across multiple model backends at a configured ratio. A team can send 50% of requests to one model and 50% to another, collecting Prometheus metrics on latency, token usage, and error rates. This produces controlled, repeatable benchmark data without building custom splitting infrastructure.
Skip if:
Skip if your benchmark requires frontier models accessible only via commercial APIs that cannot be self-hosted.
Platform engineers embedding custom routing logic in Rust services
The switchyard-libsy crate lets a platform team embed routing algorithms in an existing Rust proxy or agent runtime without adopting a new HTTP server. The library makes the routing decision and hands the model call back to the caller, so it integrates without displacing existing infrastructure.
Skip if:
Skip if your platform is not Rust-based. The library path requires a Rust application; the launcher and server paths do not, but they add an external service dependency.
Teams using escalation routing to reduce strong-model API calls
Switchyard's escalation routing runs every request on a weak (cheaper) model first, then uses a judge to decide whether the response quality is sufficient or the request must be escalated to a strong model. This reduces expensive model calls to the subset that actually needs them.
Skip if:
Skip if your workload requires the strong model for every request, or if the latency added by running a first-pass weak model is unacceptable in your pipeline.
The problem it solves#
Coding agents speak one LLM API format, but model backends speak another. An agent built for the Anthropic Messages API cannot directly call a vLLM endpoint, which serves the OpenAI Chat format. The translation layer is custom code that teams must write and maintain per deployment, and it breaks whenever either side updates its API.
The pain deepens when you want to compare multiple models or route requests intelligently. Running all traffic through a single commercial API provider means you cannot A/B benchmark open-source models at realistic traffic levels without building your own routing infrastructure. Teams either stay on expensive commercial APIs or invest significant engineering time building a proxy from scratch.
How it solves it#
Protocol translation between OpenAI and Anthropic formats
Converts requests between OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages formats. A client that speaks the Anthropic Messages API can reach an Ollama or vLLM backend that serves OpenAI Chat Completions, with no changes to the client. The translation covers requests, responses, and streaming.
Four configurable routing strategies
Supports random splitting for A/B tests and baselines, LLM-as-classifier routing where request content selects the model tier, signal-driven stage routing based on conversation signals like tool results and errors, and escalation routing that runs a weak model first and escalates based on answer quality. Each strategy is configured via route type in routes.toml.
Prometheus operational metrics
Exposes Prometheus metrics covering request counts, error rates, latency, token usage, and routing overhead. Standard Prometheus scrape endpoint means the metrics plug directly into Grafana or any existing observability stack without custom instrumentation.
Launcher mode for coding agents
Wraps coding agent invocations (Claude Code, Codex CLI, OpenClaw) to redirect their API calls through the Switchyard proxy. The agent keeps its native API format; Switchyard translates and routes to whatever open-source model backend you configure. Installed as a Python tool via uv with no changes to the agent itself.
Embeddable routing library (switchyard-libsy)
The switchyard-libsy crate embeds routing algorithms in a Rust application without owning an HTTP stack. An algorithm decides which backend to use and hands every model call back to the caller, so it drops into an existing proxy, gateway, or agent runtime. Pair with switchyard-llm-client when you want the HTTP calls handled automatically.
Strengths and trade-offs#
Strengths
- Apache-2.0 license with permissive commercial useApache-2.0 allows use in commercial products and services without copyleft requirements. Unlike AGPL (which requires distributing source changes when running a modified version as a service), Apache-2.0 lets teams embed or deploy Switchyard internally without open-sourcing their modifications.
- Active development under NVIDIA NeMoSwitchyard is maintained by the NVIDIA NeMo team and receives regular pushes. The repository was created in May 2026 and has seen continuous commits since then, with NVIDIA NIM listed as a supported backend, making it a natural fit for teams already using the NVIDIA AI ecosystem.
- Three deployment modes for different integration depthsTeams can adopt Switchyard at different integration depths: the launcher path wraps an agent invocation without running a persistent server; the server path runs a standalone Rust proxy with a TOML config; and the library path embeds routing algorithms directly into a Rust application. One project, three entry points.
- Agents require no reconfiguration to use open-source backendsBecause Switchyard translates between API formats transparently, a coding agent like Claude Code keeps calling the Anthropic Messages API, and Switchyard routes those calls to an open-source model backend. There is no agent-side code change or API key swap; only the proxy config changes.
Trade-offs
- -Pre-alpha: not for production useThe README explicitly labels Switchyard experimental and not for production use. The API and routing algorithms are expected to change significantly before v1.0. Teams with stability requirements should treat this as a development and benchmarking tool only, not a production infrastructure component.
- -Server path requires the Rust toolchainInstalling switchyard-server requires a working Rust toolchain with Cargo, which adds setup overhead compared to tools distributed as Docker images or prebuilt binaries. Teams without Rust experience will need to install the toolchain before they can run the standalone server. No Docker path is documented in the README.
- -Client-side format support limited to OpenAI and Anthropic schemasThe server accepts three client-side formats: OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. The README does not document support for other LLM API schemas. Teams whose tooling targets other provider formats cannot use Switchyard as-is without additional translation.
Switchyard vs alternatives#
Switchyard vs Commercial LLM API Providers
Most coding agents are designed to call commercial API providers directly: Claude Code calls Anthropic's API, Codex calls OpenAI's API. Every token is billed, and you cannot control model selection at runtime. Switchyard sits between the agent and the backend, translating API formats and routing to open-source models like vLLM, NVIDIA NIM, or Ollama instead.
| Feature | Switchyard | Commercial APIs |
|---|---|---|
| License | Apache-2.0 | Proprietary |
| Self-hosting | Yes | No |
| Token billing | None (your infrastructure) | Per-token |
| Protocol translation | OpenAI and Anthropic | Single format per provider |
| Multi-backend routing | Yes | No |
Switchyard is the better choice when you want to run coding agents against open-source models on your own infrastructure, control routing behavior at request time, or benchmark multiple models without per-token costs. Commercial APIs remain the better fit when you need the latest frontier models, uptime guarantees, or no infrastructure management overhead.
Switchyard vs OpenRouter
OpenRouter is a commercial API aggregation service that provides access to many models through a unified API. Like Switchyard, it abstracts model selection behind a single endpoint. Unlike Switchyard, OpenRouter is a managed service with no self-hosting option; Switchyard runs on your own servers.
OpenRouter does not perform API format translation between OpenAI and Anthropic schemas; Switchyard does. For teams who need Anthropic Messages format requests to reach an Ollama or vLLM backend, Switchyard handles the translation that OpenRouter does not.
Switchyard is the better fit for teams with infrastructure capacity who want routing logic under their own control and need cross-format translation. OpenRouter is the better fit for teams who want access to many hosted models without managing servers or writing routing configuration.
Install and self-host#
Install via the Python CLI (requires uv) or as a standalone server (requires Cargo).
```bash
uv tool install --python 3.10 "nemo-switchyard[cli]"
cargo install --locked switchyard-server
```What it's built on#
- Languages
- PythonRust
FAQ#
Is Switchyard production-ready?
No. The project README explicitly marks Switchyard as pre-alpha, experimental software not for production use. The API and routing algorithms are expected to change significantly before v1.0. Use it for development environments, research, and benchmarking only.
Which LLM backends does Switchyard support?
Switchyard routes to any OpenAI-compatible endpoint. The README specifically names vLLM, NVIDIA NIM, and Ollama as supported backends. Each backend is configured in a routes.toml file as a target. You can configure multiple backends and let Switchyard's routing algorithms choose between them at request time.
Which coding agents work with Switchyard's launcher mode?
The README names three: Claude Code, Codex CLI, and OpenClaw. Any agent that calls the OpenAI Chat Completions or Anthropic Messages API can be routed through Switchyard without code changes, since Switchyard presents itself as a compatible API endpoint to the agent.
What routing strategies does Switchyard support?
Four strategies: random routing for traffic splits and A/B tests; LLM classifier routing where request content selects the model tier via an LLM call; signal-driven stage routing that routes based on conversation signals such as tool results and error responses; and escalation routing that runs a weak model first and escalates to a strong model when the judge deems the first response insufficient.
How do I install Switchyard?
Two paths are documented. For the Python CLI launcher: install uv, then run uv tool install --python 3.10 "nemo-switchyard[cli]". For the standalone Rust server: install the Rust toolchain with Cargo, then run cargo install --locked switchyard-server. The library path adds switchyard-libsy as a Cargo dependency in your own Rust project.
Similar open-source tools#
Embabel
Agentic AI framework for the JVM
DwarfStar (ds4)
Local inference engine for DeepSeek V4 Flash/PRO and GLM 5.2
Reasonix
Open-source AI coding agent you can leave running locally.
AirLLM
Run huge LLMs on low-end GPUs with minimal VRAM
LMCache
Accelerate AI applications with caching technology
Agent-Reach
Give agents local web and social-source access