Routerly

Open source alternative to Portkey, Helicone, Bifrost and Kong AI Gateway

107 stars
AGPL-3.0
Active this week
Last commit today

Last verified Nov 5, 2026

Run a self-hosted LLM gateway that routes requests across OpenAI, Anthropic, Gemini and Ollama with 11 composable policies and no database.

Visit website GitHub107 Deploy on Hostinger

Read the full breakdownWho it is for, how it works, trade-offs, the quick start and the FAQ

who it's for

Who Routerly is for#

Platform engineers building multi-tenant AI products

Routerly's project isolation model assigns each tenant a router token with its own model list and monthly spend cap. The gateway enforces the cap before any request reaches a provider, so a single over-spending tenant cannot affect others. No application code changes are needed to onboard a new tenant.

Skip if:

If your tenants need dedicated server isolation rather than token-level isolation, running a separate Routerly instance per tenant may be more appropriate than sharing one deployment.

Developers evaluating or switching AI providers

Routing your app through Routerly means model changes happen in the dashboard, not in code. The A/B testing feature splits live traffic between two router configurations, so you can compare a cheaper model against the current one on real production requests and see cost, latency, and error rate side by side before committing.

Skip if:

If you use a single provider and have no plans to change models, running a gateway adds latency and infrastructure overhead with no benefit.

Teams centralizing access to Cursor, Open WebUI, or similar AI tools

Routerly accepts connections from any tool that speaks the OpenAI or Anthropic format. Configure Cursor, Open WebUI, LibreChat, or LangChain to point at Routerly instead of the provider directly, then apply guardrails, budget caps, and PII scrubbing at the gateway level for the whole team from one dashboard.

Skip if:

If your AI tools are SaaS-hosted and cannot be configured to use a custom API endpoint, local gateway routing will not reach their traffic.

Cost-conscious developers running local and cloud models together

Routerly can register both local Ollama models and cloud models in a single router. The built-in `cheap` routing profile sends simple requests to the free local model and falls through to a cloud model for complex ones. The dashboard cost breakdown shows how much each routing decision saves compared to always using the cloud path.

Skip if:

If you only need local model access without routing to cloud providers, running Ollama directly is simpler and has no overhead.

the problem

The problem it solves#

Applications that integrate AI providers directly absorb every reliability and cost risk of that choice. When a provider's API degrades, the application degrades with it. Pricing changes land on the bill with no warning. And actual AI spend only surfaces on the monthly invoice, not in per-call data that could inform routing decisions.

The harder challenge is cost optimization at the per-request level. Sending every request to the most capable model wastes spend on simple tasks; routing everything to the cheapest model hurts quality for complex ones. Most teams handle this with hand-written conditional logic that breaks every time a provider releases a new model, changes pricing, or adjusts their context limits.

how Routerly solves it

How it solves it#

11 Composable Routing Policies

Policy scoring runs on every request in parallel. The `cheapest` policy minimizes cost, `health` deprioritizes failing endpoints, `llm` calls a language model to pick the best candidate for the specific prompt, and `semantic-intent` classifies the request with embeddings. Any combination of policies stacks into one ranking per request, and a failed call automatically falls through to the next candidate.

Drop-in Compatibility with OpenAI and Anthropic Formats

Routerly implements the OpenAI (`/v1/chat/completions`, `/v1/responses`) and Anthropic (`/v1/messages`) wire formats with full streaming. Switching an existing application means changing one URL: no SDK changes, no code changes. AI tools and frameworks that speak either protocol, including Cursor, LangChain, Open WebUI, and the Anthropic SDK, connect by pointing at the Routerly address.

Budget Enforcement and Per-Token Cost Tracking

Every request is priced per token using built-in presets for GPT-4o, Claude, and Gemini, with custom pricing available for any model. Hard spend caps at the router or global level block over-budget requests before they reach the provider, with limits configurable per hour, day, week, or month.

No Database Required

Configuration lives in plain JSON files under `~/.routerly/`, with no Postgres, no Redis, and no migration steps. The gateway stores API keys encrypted and router tokens as SHA-256 hashes; a Docker volume holds the data directory. A single `docker run` command produces a fully operational gateway with no external services to provision.

Multi-Tenant Project Isolation

Each project gets a separate bearer token with its own model list, routing configuration, and budget limits. One instance of Routerly can serve multiple teams or tenants; there is no cross-project request or budget visibility. Per-project budget caps enforce spending at the router level before any call leaves your infrastructure.

8 Opt-in Token Optimizers

Eight token optimizers reduce prompt size before each call. `session-dedup` removes duplicate context from long sessions, `ccr` compresses redundant content, and `llmlingua-2` applies learned compression. Each optimizer is off by default and enabled per router, so you can run them on high-volume routes without affecting others.

strengths · trade-offs

Strengths and trade-offs#

Strengths

LLM-Powered RoutingThe `llm` and `semantic-intent` routing policies are unique among self-hosted gateways. Among the gateways in Routerly's README comparison table (LiteLLM, Bifrost, Portkey's OSS gateway), only Routerly documents an AI-powered routing policy that adapts decisions to the content of each request at runtime, accounting for prompt complexity and task type rather than just cost and latency signals.
No Infrastructure Dependencies Beyond DockerMost LLM proxy setups require Postgres for spend tracking and Redis for caching or rate limiting. Routerly stores everything in JSON files, which means a single `docker run` command produces a fully operational gateway with per-token billing, budget enforcement, and a web dashboard. No database to provision, no migration to run.
Native Anthropic Messages API FormatRouterly serves native Anthropic Messages API requests at `/v1/messages`, not through a translation layer. The Python and Node.js Anthropic SDKs connect by changing `base_url`; full streaming and tool_use responses are supported. This matters for teams using tools built on the Anthropic SDK, including Claude Code and OpenClaw, that cannot easily switch to the OpenAI format.
Transparent Per-Decision Routing TracesEvery routing decision records the policy scores that led to the choice and the model selected. The dashboard Playground lets you replay a request and inspect the full trace: which policies ran, what each scored, and which model won. This makes it straightforward to debug routing behavior and tune policy weights without reading logs.

Trade-offs

-AGPL-3.0 License Restricts Hosted ResaleYou can run Routerly for yourself, your team, or your clients on your own servers at no cost. If you modify the source and offer Routerly as a hosted service to others (a managed or cloud service), AGPL-3.0 requires publishing your modifications. Teams running it privately or as internal infrastructure are unaffected; this restriction only applies to reselling managed access to others.
-SSO Not Yet AvailableDashboard access uses local user accounts. OAuth 2.0, OIDC, and LDAP login are on the roadmap but not shipped as of v0.5.3. Teams whose security policies require SSO for new services will need to manage accounts manually or wait for that feature before deploying organization-wide.
-Early-Stage Project with a Growing EcosystemRouterly has 107 GitHub stars and 9 open issues as of November 2026. It is actively maintained (last push November 4, 2026) and ships a substantial feature set, but as a newer project it has a smaller ecosystem of community guides and third-party integrations compared to LiteLLM. API stability, documented upgrade paths, and production war stories are still accumulating.

versus alternatives

Routerly vs alternatives#

Routerly vs Portkey

Portkey provides an LLM gateway as a managed cloud service, with an MIT-licensed open source proxy for self-hosters and paid hosted or enterprise tiers for teams that want managed observability. Routerly is fully self-hosted, which means routing, cost tracking, and usage data stay on your own infrastructure rather than passing through a third-party service.

Feature	Routerly	Portkey
License	AGPL-3.0	MIT (self-hosted proxy) / commercial (hosted/enterprise)
Self-hosted	Yes	Yes (proxy tier)
Routing policies	11 composable policies	Fallbacks, retries, weighted load balancing
LLM-powered routing	Yes	Not documented
No database required	Yes	n/a (managed service)
PII scrubbing	Yes (all tiers)	Enterprise tier

Routerly is the right choice when routing decisions need to stay on your own servers and you want the full policy suite including LLM-powered routing. Portkey's paid tiers suit teams that want managed observability dashboards, shared team access, and support without running infrastructure.

Routerly vs Bifrost

Bifrost is an Apache-2.0 open source LLM gateway. Routerly's README comparison (verified October 2026 against each project's public documentation) shows both support native Anthropic API format and self-hosting. Bifrost does not document LLM-powered or intent-based routing, and its enterprise tier is required for some governance features that Routerly includes in the open tier.

Feature	Routerly	Bifrost
License	AGPL-3.0	Apache-2.0 (enterprise tier exists)
Native Anthropic API	Yes	Yes (`/anthropic`)
LLM-powered routing	Yes	Not documented
No database for core use	Yes (JSON files)	Not verified
Web dashboard	Yes	Yes
Guardrails	Yes (open tier)	Enterprise tier

Routerly is the better fit when you need the full routing policy suite or guardrails without a commercial enterprise tier. Bifrost is worth evaluating if your project requires an Apache-2.0 license for open source compatibility and cannot use AGPL.

install · quick start

Quick start#

bash

Self-host Routerly on Docker with a persistent data volume.

```bash
docker run -d \
  --name routerly \
  -p 3000:3000 \
  -v routerly_data:/data \
  -e ROUTERLY_HOME=/data \
  --restart unless-stopped \
  inebrio/routerly:v0.5.3
```

tech stack · detected from GitHub

What it's built on#

Languages: JavaScriptTypeScript
Frameworks: React

frequently asked

FAQ#

Is Routerly free to use?

Yes. Routerly is AGPL-3.0 licensed and free to run on your own infrastructure. There is no paid tier or usage fee for self-hosting. If you modify the source and offer Routerly as a hosted service to others, AGPL-3.0 requires publishing those modifications; running it for yourself or your team, including commercially, has no such requirement.

Does Routerly work with the Anthropic SDK?

Yes. Routerly serves the Anthropic Messages API at /v1/messages, so the Python and Node.js Anthropic SDKs connect by changing base_url to the Routerly address. Full streaming and tool_use responses are supported. AI tools built on the Anthropic protocol, including Claude Code and OpenClaw, work by pointing their custom endpoint setting at the same address.

How does Routerly handle AI provider outages?

When a provider returns an error or times out, Routerly falls back to the next highest-scoring candidate model automatically. The health routing policy deprioritizes endpoints with recent errors, so a degraded provider is routed around on subsequent requests without manual intervention. Registering the same capability across multiple providers (OpenAI, Anthropic, Gemini) gives Routerly candidates to route to when one is down.

What data does Routerly store, and where?

Configuration and usage data live in JSON files in the ~/.routerly/ directory (or the Docker volume mounted at /data). API keys are stored encrypted; router tokens are stored as SHA-256 hashes; passwords use bcrypt. No data leaves your server unless you configure a notification channel like Slack or a webhook.

Does the `llm` routing policy add latency to every request?

Yes, when enabled, the llm policy makes an additional AI call to pick the model, adding the time of that call to each request. Teams sensitive to latency can disable it and use deterministic policies like cheapest, health, or capability instead. These run in-process with millisecond overhead. The llm policy is off by default and opt-in per router.

also worth a look