Who leanctx is for#
Backend engineers running RAG pipelines with large documents
leanctx routes retrieved passages through LLMLingua-2, trimming each document by roughly 40% before injection into the LLM request. On prose-heavy traffic (retrieved docs, PDFs, meeting transcripts), blended savings reach 36.7%. Code in tool results stays verbatim; only the document text compresses. The N=503 benchmark showed ~$78 saved per 1,000 requests at Sonnet input pricing on a 45.8% prose corpus.
Skip if:
Your retrieved documents are already short or pre-summarized. If average retrieved context is under 500 tokens per request, the compression overhead and the 1.2 GB model download are unlikely to pay off.
LangChain and LangGraph agent developers managing growing tool-call histories
Long-running agents accumulate tool outputs and conversation turns across many requests. leanctx compresses prior conversation history and log dumps while preserving tool_use_id, tool names, and tool inputs byte-for-byte. A 9-message coding agent transcript (~2.1K tokens) showed 35.6% reduction while keeping all tool linkage and code blocks untouched.
Skip if:
Your agent loop is short (under 5 turns) or you already truncate history before each call. For very short conversations, compression overhead does not justify the savings.
LLMOps engineers optimizing production inference costs
leanctx emits OpenTelemetry spans and metrics per call, labeled by provider, method, and status. You can correlate its output with provider-side cache-hit rates in the same dashboard. The SelfLLM cross-provider benchmarks show 41.6-49.1% compression rates using cheap model tiers, at $0.0001-$0.0016 per compression call depending on provider.
Skip if:
Your primary cost driver is output tokens rather than input tokens. leanctx only compresses the request (input) side; it has no effect on generation length or output token counts.
Teams switching away from hosted compression APIs
If your current setup sends prompts to a third-party compression service like The Token Company, leanctx provides an on-premise alternative that runs inside your existing Python deployment with no additional server to operate. Compression latency is 47ms p50 on GPU for the sidecar path, with invariant checks verifying structural integrity before each upstream call.
Skip if:
You need compression for non-Python workloads (Node.js, Go, Java). leanctx is Python-only; the ClawRouter TypeScript sidecar adds server complexity that may outweigh the benefit.
The problem it solves#
Input token costs become a real budget concern when you are running retrieval-augmented generation with large documents, maintaining long agent conversation histories, or processing multi-step tool call chains that accumulate across turns. A RAG app that injects five 3,000-token documents per query at Sonnet pricing is paying for most of that context on every call, even when the same documents appear across many requests.
The two standard mitigations both have limits. Provider prompt caching covers stable prefixes like system prompts and static document pools, but it does not help when retrieved content is freshly fetched per query. Naive truncation discards context from the middle of documents, which is where many answers live, as the LongBench v2 benchmarks make concrete. Knowing which parts of a live request can survive compression, and which must remain untouched, is harder than shortening text.
How it solves it#
Loss-tolerance routing
Classifies every segment of an outbound LLM request into one of three categories: zero tolerance (code, stack traces, tool_use_id, tool call names and inputs), high tolerance (prose, retrieved passages, conversation history), and conditional (low-confidence prose, oversized context). Each category routes to a different compressor, so structural content is never altered regardless of overall compression settings.
On-device LLMLingua-2 compression
The Lingua route uses LLMLingua-2, an extractive model that runs locally inside your Python process at zero marginal cost per call. On the N=503 LongBench v2 sweep, it compressed eligible prose segments by 40.8% (blended savings of 18.7% on a 45.8% prose corpus). No data leaves your infrastructure during compression.
Drop-in SDK wrapper
Wraps the OpenAI, Anthropic, and Gemini Python clients with the same interface. Switching from `from openai import OpenAI` to `from leanctx import OpenAI` is the only required code change; all existing API call patterns, parameters, and response shapes continue to work unchanged.
Fail-open invariant checks
After compression, leanctx checks message count and ordering, every tool_use_id, tool name and input, and code inside tool results. Any check failure, timeout, or sidecar error sends the original uncompressed request to the provider. A compression outage costs savings, never availability.
OpenTelemetry observability
Emits spans and metrics for every compression call when enabled via `leanctx_config["observability"]["otel"]`. Each root span records provider, method, input and output tokens, cost in USD, and duration. Five labeled metrics (4 counters, 1 histogram) give per-provider and per-method breakdowns without the library owning the OTel SDK.
Reproducible benchmark CLI
Ships seven named scenarios via `leanctx bench run`. Each scenario runs the real LLMLingua-2 model and asserts named invariants, with versioned JSON output. The `agent-structural` scenario checks byte-identical preservation of code blocks, tracebacks, and tool linkage and exits non-zero on any regression, making it suitable as a CI gate.
Strengths and trade-offs#
Strengths
- Prompts never leave your infrastructureUnlike The Token Company and other hosted compression APIs, leanctx runs the compression model locally inside your Python process. No outbound calls go to third-party compression endpoints, and no prompt data is transmitted to external servers. The LLMLingua-2 model loads from a local Hugging Face cache on first use.
- Composes with provider cachingleanctx and provider-side prompt caching are complementary. Provider caching covers stable prefixes (system prompts, tool definitions, static document pools) at up to 90% discount on cached reads. leanctx covers the dynamic suffix that changes every call. You mark your stable prefix with cache_control and let leanctx handle the variable portion; both savings stack.
- Independently verified benchmarksThe N=503 LongBench v2 sweep was executed and audited by outside contributors, not the maintainer. An earlier draft claiming +7.4% on long context was caught as evaluation noise (McNemar p=0.143), and two methodology fixes followed before results were published. Per-item records are committed to the repo and independently reproducible from a clean checkout.
- MIT license, zero marginal cost per Lingua callMIT-licensed with no vendor lock-in. The Lingua route uses the on-device LLMLingua-2 model at zero cost beyond compute; there is no per-call fee and no subscription. The SelfLLM route calls a provider API you configure at your own discretion. No third-party compression billing is introduced.
Trade-offs
- -First Lingua run downloads ~1.2 GB of model weightsThe first call through the Lingua route downloads the LLMLingua-2 xlm-roberta-large-meetingbank model weights (~1.2 GB) to the Hugging Face cache. Subsequent calls reuse the cache, but cold-start setup time is meaningful on constrained environments, thin CI machines, or slow network connections. Omitting the `[lingua]` extra leaves leanctx in passthrough mode with no download required.
- -Accuracy cost on compressed contentContent routed through LLMLingua-2 in the N=503 LongBench v2 sweep saw a 3.9 percentage point accuracy drop (43.4% to 39.5%). For short/Lingua items specifically (N=68), the drop was 17.6 pp. The overall 1.8 pp hit is bounded by the 54.2% of tokens that route verbatim, but accuracy loss on the compressed half is real and task-dependent. Test on a representative sample of your own workload before enabling in production.
- -Python-only SDKleanctx is a Python SDK. Applications written in TypeScript, Go, Java, or other languages have no equivalent in-process client. The ClawRouter integration provides a TypeScript connector via a sidecar architecture, but that adds operational complexity compared to the in-process Python approach.
- -Young project with early-stage APIThe repo was created in April 2026, has 319 stars, and 3 forks. It is an early-stage library with a single open issue at the time of writing. The API surface may change across minor releases, and long-term maintenance continuity is not yet established.
leanctx vs alternatives#
leanctx vs The Token Company
The Token Company is a hosted compression API for LLM prompts. leanctx covers the same use case but takes a different architecture: compression runs inside your Python process rather than on an external server.
| Feature | leanctx | The Token Company |
|---|---|---|
| License | MIT | Proprietary |
| Compression location | In-process (local) | Hosted servers |
| Prompt data leaves your infra | No | Yes |
| Compression model | LLMLingua-2 (open source) | Closed-source |
| Self-hosting | Yes (Python package) | No |
leanctx is the better choice when your prompts contain sensitive user data that cannot leave your infrastructure, when you need to audit or modify the compression logic, or when you want to avoid per-call billing for a high-volume application. The on-device Lingua route has zero marginal cost per call once the model is cached locally, and the MIT license imposes no resale or usage restrictions on internal team use.
The Token Company is worth considering if you need compression without a Python runtime, want a fully managed service with no model weight download to maintain, or your application is written in a language other than Python. For TypeScript, Go, or Java stacks where a Python sidecar adds operational overhead, a hosted API may be the more pragmatic starting point.
Install and self-host#
Install leanctx via pip with extras for your LLM provider and the Lingua compression backend.
```bash
pip install 'leanctx[openai,lingua]'
```What it's built on#
- Languages
- JavaScriptPythonTypeScript
- Frameworks
- FastAPILangChainLangGraph
FAQ#
Does leanctx send my prompts to a third-party compression server?
No. The Lingua route runs LLMLingua-2 locally inside your Python process; the model weights download to your local Hugging Face cache (~1.2 GB on first use). The SelfLLM route calls the LLM provider API you configure, not an intermediate service. No third-party compression endpoint receives your data by default.
How much do token savings vary by workload type?
Savings scale with the prose share of your traffic. Code-heavy agent traffic (25% prose) saves roughly 10.2%; the LongBench v2 benchmark corpus (45.8% prose) saves 18.7%; prose-heavy workloads like document Q&A or transcripts (90% prose) save up to 36.7%. The blended formula documented in the repo is: savings = prose_token_share x 40.8% (95% CI 37.8-43.7%).
What happens if the compression step fails or times out?
leanctx fails open. After compression, it checks message count and ordering, every tool_use_id, tool name and input, and code inside tool results. Any check failure, timeout, or unreachable sidecar sends the original uncompressed request to the provider. A compression outage costs savings, not availability.
Is leanctx compatible with LangChain or LangGraph?
Yes. leanctx wraps the OpenAI, Anthropic, and Gemini Python clients at the SDK level, so any LangChain or LangGraph application using those providers works without additional changes. The project's GitHub topics explicitly list langchain and langgraph as supported integration targets.
How does leanctx compare to provider-side prompt caching?
The two are complementary, not competing. Provider prompt caching (Anthropic cache_control, OpenAI, Gemini) applies up to a 90% discount on stable prefixes like system prompts and static document pools, but it does not help with dynamic per-query content. leanctx compresses that dynamic suffix. Mark your stable prefix with cache_control and let leanctx handle the variable portion; both savings stack independently.
Similar open-source tools#
headroom
Compress LLM context before it reaches the model
OpenCode
OpenCode is an open-source AI coding agent that assists developers in
token-optimizer-mcp
Token savings and persistent knowledge graphs for AI coding agents
llama-swap
Hot-swap AI models on your local inference server
magnitude
Local models for your agent, tuned for your hardware
sie
One self-hosted cluster for all the models your agents need