Who Tokenspeed is for#
AI infrastructure teams self-hosting frontier model inference
Teams running inference clusters for models like Kimi K3, DeepSeek V4, or Qwen3.8 on NVIDIA GB200/GB300 or AMD Instinct hardware can deploy TokenSpeed to maximize throughput per GPU. The kernel plugin system allows tuning for specific model architectures and accelerator generations.
Skip if:
Teams without NVIDIA Blackwell or AMD Instinct accelerators, or teams that need a fully managed inference service rather than a self-hosted engine.
Agentic platform developers serving high-concurrency coding agents
Systems generating coding-agent traffic produce contexts over 50K tokens, dozens of conversation turns, and per-user latency requirements around 70-200 TPS. TokenSpeed's scheduler and MLA implementation are designed for this traffic pattern rather than generalized chatbot serving.
Skip if:
Teams building simple chatbots or single-turn applications where standard inference engines already meet latency requirements.
ML researchers and kernel engineers iterating on attention implementations
The pluggable kernel system with its portable public API and centralized registry makes TokenSpeed an environment for writing and benchmarking custom attention kernels. The kernel layer is designed for rapid iteration and includes a documented path for contributing heterogeneous accelerator plugins.
Skip if:
Researchers focused on model training or fine-tuning. TokenSpeed is an inference-only engine with no training path.
Organizations evaluating TensorRT-LLM alternatives for Blackwell deployments
Teams currently using TensorRT-LLM for Blackwell deployments can evaluate TokenSpeed using the published side-by-side benchmarks on Kimi K2.5 SWE-smith traces. The MIT license and non-profit LightSeek Foundation governance differ substantially from NVIDIA's proprietary NIM stack.
Skip if:
Teams with existing TensorRT-LLM deployments meeting performance targets where migration cost outweighs architectural or governance benefits.
The problem it solves#
Production LLM inference for agentic workloads runs on different constraints than standard chatbot serving. Coding agents generate contexts that routinely exceed 50K tokens and conversations that span dozens of turns, all while users expect fast output, typically 70 tokens per second per user and often 200 or higher. Meeting that latency floor at high concurrency requires careful control over KV cache allocation, batch scheduling, and kernel performance. Most general-purpose inference engines were designed around single-turn or short-context workloads and did not treat these requirements as the primary design target.
A second challenge is hardware heterogeneity. Production inference fleets run on NVIDIA and AMD accelerators, and teams tuning performance for specific frontier models need access to the kernel layer. Closed-source inference services offer no path to kernel-level inspection or modification. Teams serving MoE models like DeepSeek V4 or Kimi K3 on their own infrastructure often find that general-purpose engines leave throughput per GPU unrealized because agentic traffic's KV cache access patterns differ from what those engines optimized for.
How it solves it#
C++ FSM Scheduler with Python Execution Plane
The control plane runs as a C++ finite-state machine, encoding request lifecycle, KV cache ownership, and overlap timing as FSM transitions. Safe KV resource reuse is enforced at compile time through the type system rather than at runtime. The execution plane stays in Python for faster feature iteration and lower cognitive load during research and development.
Pluggable Layered Kernel System
Kernels are separated from the engine core and treated as a first-class modular subsystem with a portable public API, centralized registry, and extensible plugin mechanism. This allows adding or replacing kernels for different accelerators without modifying the engine core, and supports heterogeneous hardware including NVIDIA Blackwell and AMD Instinct.
Optimized MLA for NVIDIA Blackwell
TokenSpeed includes one of the fastest Multi-head Latent Attention implementations for NVIDIA Blackwell for agentic workloads. The decode kernel folds the query-sequence axis into the head axis to improve Tensor Core utilization, and the binary prefill kernel uses a fine-tuned softmax implementation. The TokenSpeed MLA has been adopted upstream by vLLM.
Compiler-Backed Parallelism with Local SPMD
The modeling layer uses a local SPMD design. Developers specify I/O placement annotations at module boundaries, and a lightweight static compiler automatically generates the required collective operations at model construction time, eliminating manual communication logic for tensor parallelism across multiple GPUs.
SMG-Integrated AsyncLLM Entrypoint
Request handling on the CPU side uses an AsyncLLM interface integrated with SMG, providing a low-overhead entrypoint that keeps CPU-side request queueing separate from GPU execution. This is the API surface that external orchestrators, including NVIDIA Dynamo, integrate with.
Day-0 Support for Frontier Models
TokenSpeed has published day-0 launch support for Kimi K3, Kimi K2.5, DeepSeek V4, Qwen3.8, Qwen3.8-Flash-Next, GLM 5.3 Flash, TML Inkling, and Minimax M2.7. It supports both MoE and dense model architectures, NVFP4 quantization, and VLM inference.
Strengths and trade-offs#
Strengths
- Non-Profit Foundation, Vendor-Neutral GovernanceTokenSpeed is developed by the LightSeek Foundation, a non-profit, with hardware and engineering support from NVIDIA, AMD, and model teams. Unlike inference engines built by commercial API providers, there is no incentive to steer users toward a hosted service. The README describes it as the 'Switzerland' of open source inference engines, governed by a non-profit with no commercial hosting service of its own.
- Compile-Time Correctness Guarantees in the SchedulerThe C++ FSM control plane enforces safe resource management, including KV cache state, through the type system at compile time. The PyTorch Foundation named this separation of control and execution planes as TokenSpeed's primary architectural differentiator. Correctness is a property of the system structure rather than a convention that runtime checks try to uphold.
- Benchmarked Above TensorRT-LLM on Coding-Agent WorkloadsFor Kimi K2.5 on NVIDIA B200, the best configuration (Attention TP4 + MoE TP4) is roughly 9% faster at batch size 1 and roughly 11% higher in throughput at 100 TPS/user for coding-agent workloads above 70 TPS/user. These figures come from LightSeek's published benchmarks using SWE-smith traces, which mirror production coding-agent traffic.
- MIT License, No Usage Restrictions for Self-HostersThe MIT license lets teams run, modify, and redistribute TokenSpeed on their own infrastructure without licensing fees, usage metering, or service agreements. There are no commercial restrictions for self-hosted deployments and no dependency on any cloud provider's control plane.
Trade-offs
- -Requires High-End NVIDIA or AMD GPU HardwareTokenSpeed targets NVIDIA Blackwell (B200, GB200, GB300) and AMD Instinct (MI355X, MI455X) accelerators. It is not designed for CPU-only inference or commodity cloud GPUs. Teams without access to these accelerator generations cannot run the engine at its intended performance envelope.
- -Young Project Still Undergoing Production HardeningDevelopment began in mid-March 2026 and the project was announced in May 2026. As of October 2026, the engine and kernels remain under active development with production hardening ongoing. With 99 open GitHub issues, teams adopting it for production workloads should track upstream changes closely.
- -PD Disaggregation Not Yet in the Main DocumentationThe published performance benchmarks focus on single (non-disaggregated) deployment only. Prefill-decode disaggregation support is under development, and the LightSeek team has indicated coverage in a separate follow-up. Teams that need PD disaggregation in production should treat this as a near-term gap.
Tokenspeed vs alternatives#
TokenSpeed vs Fireworks AI
Both target fast LLM inference, but in different deployment models. Fireworks AI is a managed API with no self-hosting option; TokenSpeed is an open source engine you run on your own GPU hardware.
| Feature | TokenSpeed | Fireworks AI |
|---|---|---|
| License | MIT | Proprietary |
| Self-hosting | Yes | No |
| GPU support | NVIDIA Blackwell, AMD Instinct | Managed fleet |
| Pricing | Free (your hardware costs) | Per-token API fees |
| Governance | Non-profit (LightSeek Foundation) | Commercial |
TokenSpeed is the better choice when you run your own inference cluster, need kernel-level control for specific models, or serve high-volume agentic workloads where per-token API costs add up. Fireworks AI is the better choice when you need a managed service with SLA guarantees, do not have GPU hardware, or need broad model availability without infrastructure management.
TokenSpeed vs Together AI
Together AI contributed engineering work to TokenSpeed and is listed in its acknowledgments, which makes the two partially complementary. Together AI is a commercial inference API; TokenSpeed is the open source engine co-developed with Together AI's inference team and published for self-hosters.
| Feature | TokenSpeed | Together AI |
|---|---|---|
| License | MIT | Proprietary |
| Self-hosting | Yes | No |
| Control | Full kernel and scheduler access | API surface only |
| Pricing | Free (your hardware costs) | Per-token API fees |
For teams that want the engineering quality behind Together AI's serving stack without the managed API layer, TokenSpeed is the direct path. Together AI remains the better choice for organizations that need managed inference without operational overhead.
TokenSpeed vs NVIDIA NIM
NVIDIA NIM provides containerized LLM inference with NVIDIA's optimization stack. TokenSpeed is an alternative for teams who need MIT-licensed, non-profit-governed inference software on Blackwell hardware rather than NVIDIA's proprietary NIM containers.
| Feature | TokenSpeed | NVIDIA NIM |
|---|---|---|
| License | MIT | Proprietary NVIDIA |
| Self-hosting | Yes | Yes (NVIDIA hardware) |
| Governance | Non-profit (LightSeek) | Commercial (NVIDIA) |
| Vendor neutrality | NVIDIA + AMD | NVIDIA only |
NIM is reasonable when you want NVIDIA's official container support and full optimization stack from the hardware vendor. TokenSpeed is worth evaluating when you need AMD hardware support, MIT license terms, or independent non-profit governance.
What it's built on#
- Languages
- C++Python
- Frameworks
- FastAPI
FAQ#
Is TokenSpeed free to use?
Yes. TokenSpeed is MIT-licensed. You can run it on your own hardware, modify it, and use it commercially without licensing fees or service agreements. The LightSeek Foundation does not offer a managed hosted version; self-hosting on your own GPU hardware is the only deployment path.
How does TokenSpeed compare to vLLM?
TokenSpeed targets higher throughput on agentic workloads specifically, with a C++ FSM scheduler that enforces KV cache resource correctness at compile time rather than runtime. Where vLLM prioritizes broad model compatibility and ease of use, TokenSpeed trades some of that generality for maximum performance on NVIDIA Blackwell and AMD Instinct hardware. The TokenSpeed MLA kernel has been adopted upstream by vLLM, so the two projects share some kernel work.
What hardware does TokenSpeed support?
TokenSpeed has published performance results for NVIDIA Blackwell B200, GB200, and GB300, and AMD Instinct MI355X and MI455X. The kernel plugin system is designed to support heterogeneous accelerators. Older NVIDIA architectures are not listed in the project's performance targets or day-0 model launch records.
What models does TokenSpeed support?
The project has day-0 support records for Kimi K3, Kimi K2.5, DeepSeek V4, Qwen3.8, Qwen3.8-Flash-Next, GLM 5.3 Flash, TML Inkling, and Minimax M2.7. Model recipes covering serving configurations are available in the LightSeek documentation.
Who maintains and funds TokenSpeed?
TokenSpeed is developed under the LightSeek Foundation, a non-profit organization. Hardware and engineering contributions come from NVIDIA DevTech, AMD Triton, Qwen Inference, Together AI, Mooncake, and others listed in the project acknowledgments. The non-profit structure means no commercial hosted inference service is driving the project's direction.
Similar open-source tools#
sie
3,410 GitHub starsOne self-hosted cluster for all the models your agents need
Alternative to
Baseten
AirLLM
39,181 GitHub starsRun huge LLMs on low-end GPUs with minimal VRAM
Alternative to
Hugging Face Inference Endpoints
Unsloth
78,176 GitHub starsTrain LLMs locally without code using a browser-based interface
Alternative to
Together AI
Embabel
4,520 GitHub starsAgentic AI framework for the JVM
Alternative to
Google Gemini Enterprise Agent Platform
InsForge
13,132 GitHub starsBackend-as-a-service platform built for AI coding agents
Alternative to
Supabase