Who sie is for#
AI engineers building production RAG pipelines
SIE serves the full retrieval stack from one cluster: embedding documents with bge-m3 or SPLADE, reranking with ColBERT or qwen3-reranker, and parsing PDFs with GLM-OCR or PaddleOCR-VL. Teams eliminate separate API billing for each step and keep all data inside their own network.
Skip if:
You are prototyping with low query volume. A single managed API call is simpler when you are not yet at the scale where per-token fees are meaningful and Kubernetes overhead is justified.
Compliance-constrained teams that cannot send data to third-party APIs
SIE runs fully within your VPC or on-premises. Prompts, documents, and embeddings never leave your network. Air-gapped installs pull models from mirrored snapshots, making it usable in classified or regulated environments.
Skip if:
Your compliance requirements permit sending data to SOC2 Type 2-certified managed services. In that case, Fireworks AI or Baseten may offer faster time-to-production with less infrastructure overhead.
Teams migrating agent code from OpenAI-compatible APIs
SIE's API matches the OpenAI spec at /v1/embeddings, /v1/chat/completions, /v1/completions, and /v1/responses. A one-line base URL change migrates most existing agent code. Python and TypeScript SDKs are available alongside the REST API.
Skip if:
Your agents depend on frontier model capabilities (GPT-5, Claude, Gemini). SIE serves open models only; it is not a proxy for frontier-lab APIs and does not match their performance on general reasoning tasks.
MLOps engineers operating multi-model inference clusters
SIE's POOL-THEN-BATCH architecture routes mixed model requests through a cluster-wide queue, improving GPU utilization from roughly 51% (worker-local) to 89%. KEDA autoscaling handles scale-to-zero and burst traffic automatically.
Skip if:
Your inference workload is a single model type at predictable volume. A dedicated model server (vLLM, TEI) is simpler to operate without SIE's orchestration layer.
The problem it solves#
Building production AI agents typically means paying multiple inference APIs simultaneously: one for embeddings, another for reranking, a third for chat completion, and more for OCR or structured extraction. Each call goes to a different third-party server, so prompts and documents leave your infrastructure with every request. At scale, per-token fees from commercial providers add up to several times the actual compute cost.
The harder problem is orchestration. Each model type needs a different runtime, so teams end up managing multiple model servers with no shared batching or resource pooling. Worker-local queue architectures run GPU resources unevenly, keeping utilization well below 70%. Teams either accept that inefficiency or spend engineering effort building custom queuing on top of existing model servers that were not designed to share GPU capacity across model families.
How it solves it#
Cluster-wide queue with 89% GPU utilization
SIE routes all inference work through a shared pool queue rather than per-worker queues. Mixed request sizes batch cleanly against the full GPU pool, reaching 89% GPU efficiency versus roughly 51% in worker-local architectures. This means fewer GPUs serve the same throughput at production volume.
OpenAI-compatible API for drop-in migration
SIE publishes /v1/embeddings, /v1/chat/completions, /v1/completions, and /v1/responses. Migrating agent code to SIE requires only a base_url change; no SDK rewrites needed. Python and TypeScript SDKs are available alongside the REST API.
Five agent task types in one server
One cluster serves search (embed, match, rerank), document-to-markdown conversion, structured JSON extraction, content safety classification, and the agent loop LLM. Key models: bge-m3, splade-v3, colbertv2, qwen3-reranker, glm-ocr, gliner2, granite-guardian-2b, and qwen3.6-27b.
Helm charts and Terraform for production Kubernetes
SIE ships Helm charts with KEDA autoscaling (scale to zero), a load-balancing gateway, and Grafana dashboards. Public Terraform modules cover AWS EKS, Google GKE, Azure AKS, and Alibaba Cloud ACK. Air-gapped installs run from mirrored model snapshots.
100+ models with on-demand loading and LRU eviction
The model catalog covers embeddings (bge-m3, Stella, SPLADE, ColBERT), rerankers, OCR engines (GLM-OCR, PaddleOCR-VL, MinerU, Docling), entity extractors (GLiNER), safety classifiers (Granite Guardian), and LLMs (Qwen3 family). Models load on first call and evict under memory pressure.
Strengths and trade-offs#
Strengths
- Apache 2.0 license with no usage restrictionsSIE is Apache 2.0 licensed across the full stack, including Helm charts and Terraform modules. Unlike managed inference APIs that bill per token, you run SIE on your own GPUs with no per-request fees and no restrictions on commercial use or modification.
- Data never leaves your infrastructureAll inference runs inside your cloud or on-premises network. Prompts, documents, and model outputs stay within your network boundary. This is essential for teams subject to GDPR, HIPAA, or internal data-classification policies that prohibit sending text to third-party APIs.
- Integrates with nine agent frameworks and vector storesSIE integrates with LangChain, LlamaIndex, Haystack, DSPy, CrewAI, Chroma, Qdrant, Weaviate, and LanceDB. The OpenAI Agents SDK is supported via a one-line base URL change, with LangGraph and CrewAI integrations noted as upcoming.
- MTEB-benchmarked model catalogEmbedding and retrieval models in the catalog are benchmarked on MTEB tasks, so you can select models by documented task performance. The catalog cites bge-m3 as 2.7x faster than Cohere rerank-3.5 on MTEB AskUbuntu and Qwen3.6-27B at 96% on the AA Intelligence Index versus GPT-5.1.
Trade-offs
- -Kubernetes expertise required for production deploymentRunning SIE at production scale requires operating a Kubernetes cluster: managing GPU node pools, Helm releases, KEDA autoscaling, and Grafana dashboards. Teams without existing Kubernetes infrastructure face significant setup time before their first production inference call.
- -GPU hardware required for high-performance inferenceMost catalog models need NVIDIA GPUs (L4 or H100 class). The CPU image runs lightweight models for development, but embedding quality and LLM generation performance degrade significantly. Cloud GPU instances add ongoing cost that only offsets per-token API savings at sustained inference volume.
- -Anonymous telemetry enabled by defaultSIE collects anonymous usage data (version, OS, architecture, GPU type) by default. Disabling it requires setting SIE_TELEMETRY_DISABLED=1 or DO_NOT_TRACK=1. No IP addresses or request data are collected, but the opt-out requirement may conflict with strict data-governance policies.
sie vs alternatives#
SIE vs Fireworks AI
Fireworks AI is a managed inference API optimized for fast open-model serving, primarily LLMs and embeddings. SIE is a self-hosted inference cluster. The decision between them comes down to where your data must live and whether your team can operate Kubernetes.
| Feature | SIE | Fireworks AI |
|---|---|---|
| License | Apache 2.0 | Proprietary |
| Deployment | Self-hosted (Kubernetes, Docker) | Managed API only |
| Data residency | Full (stays in your cloud) | Requests sent to Fireworks servers |
| Model types | LLMs, embeddings, rerankers, OCR, safety, structured output | LLMs and embeddings |
| Pricing | Compute cost only (no per-token fees) | Per-token billing |
| Air-gapped support | Yes | No |
SIE is the better choice when your data cannot leave your infrastructure, when you are running sustained inference volume where per-token fees are significant, or when you need model types beyond LLMs and embeddings (OCR, safety classifiers, entity extractors). Fireworks AI is a better fit when you need fast time-to-production, lack Kubernetes expertise, or have bursty or low-volume workloads where the overhead of running a Kubernetes cluster outweighs per-token costs.
SIE vs Baseten
Baseten is a model serving platform that handles deployment, autoscaling, and GPU management for custom and open models. SIE covers similar ground but runs entirely in your own cloud account, giving you full data residency.
| Feature | SIE | Baseten |
|---|---|---|
| License | Apache 2.0 | Proprietary |
| Deployment | Self-hosted | Managed |
| Data residency | Full (your cloud) | Requests process on Baseten servers |
| Model catalog | 100+ pre-configured open models | Custom models plus curated set |
| Terraform support | Yes (AWS, GCP, Azure, Alibaba Cloud) | Not applicable |
| Pricing | Compute cost only | Per-compute-second billing |
Baseten is well-suited for teams that want to serve custom fine-tuned models without managing infrastructure themselves. SIE is the better fit when the full catalog of agent models (embedding, reranking, OCR, safety) needs to run in a single self-hosted cluster with no data leaving your account. Teams with strict compliance requirements (GDPR, HIPAA, air-gapped environments) typically favor SIE's self-hosted deployment model.
Install and self-host#
Self-hosting uses Docker for local development and Helm for production Kubernetes clusters.
```bash
pip install "sie-server[local]" && sie-server serve
docker run --gpus all -p 8080:8080 \
-v sie-hf-cache:/app/.cache/huggingface \
ghcr.io/superlinked/sie-server:latest-cuda12-default
pip install sie-sdk
npm install @superlinked/sie-sdk
```What it's built on#
- Languages
- PythonRustTypeScript
FAQ#
Is SIE free to use?
SIE is Apache 2.0 licensed and free to self-host. The server, Helm charts, and Terraform modules are all open source with no usage fees. A managed cloud tier is listed as upcoming on the website but is not yet available; the current free path is self-hosting on your own infrastructure.
What models does SIE support?
SIE's catalog covers over 100 models across five task types: embeddings (bge-m3, SPLADE, Stella, ColBERT), rerankers (qwen3-reranker, jina-reranker), OCR (GLM-OCR, PaddleOCR-VL, MinerU, Docling), entity extractors (GLiNER), safety classifiers (Granite Guardian), and LLMs (Qwen3 family). You can also pass any Hugging Face model ID directly to the API.
How does SIE compare to Fireworks AI or Baseten?
Fireworks AI and Baseten are managed inference APIs: you send requests to their servers, pay per token or per compute-second, and they handle infrastructure. SIE runs in your own cloud account, eliminates per-token fees, and keeps all data inside your network. The tradeoff is that SIE requires Kubernetes expertise to operate at production scale, whereas managed APIs work immediately with no infrastructure setup.
Can I run SIE without a GPU?
Yes. SIE ships a CPU image that runs lighter models for development and testing. Production embedding and generation quality are lower on CPU-only hardware, and inference latency is higher. Most teams use the CPU image locally and NVIDIA GPU instances (L4 or H100 class) in production.
Does SIE work with my existing agent framework?
SIE is OpenAI-API-compatible and integrates with LangChain, LlamaIndex, Haystack, DSPy, CrewAI, Chroma, Qdrant, Weaviate, and LanceDB. A one-line base URL change in the OpenAI SDK migrates most existing agent integrations without any other code changes.
Similar open-source tools#
AirLLM
Run huge LLMs on low-end GPUs with minimal VRAM
LMCache
Accelerate AI applications with caching technology
OpenCode
OpenCode is an open-source AI coding agent that assists developers in
heretic
Decensor local LLMs automatically with directional ablation
accelerate
Distribute PyTorch training to multi-GPU and TPU with four lines
marin
Open lab for training foundation models together