Open Source Alternatives
  • Alternatives
  • Categories
    Categories by size1,283 projects
    • Developer Tools660
    • AI & Machine Learning543
    • Business & Productivity307
    • Web Development219
    • Security & Monitoring198
    • Communication & Collaboration168
    • IT Management164
    • Design & Creative142
    • LLMOps & AI Tooling140
    • DevOps & CI/CD137
    Browse all 21 categoriesCloud & Hosting, Product & Project Management and 9 more
  • Blog
  • Advertise
SearchNotion, Slack, password manager.../
Submit
Open Source Alternatives
  • Alternatives
    • Developer Tools660
    • AI & Machine Learning543
    • Business & Productivity307
    • Web Development219
    • Security & Monitoring198
    • Communication & Collaboration168
    • Browse all 21 categories
  • Blog
  • Advertise
Submit your project

New open source projects by email

The projects we add and every new guide. Free, unsubscribe any time.

Open Source Alternatives LogoOpen Source Alternatives

Handpicked Open Source Alternatives to Paid Softwares

Add Open Source Alternatives as a preferred source on Google

Product

  • Categories
  • Tag
  • Advertise

Resources

  • Blog
  • Collection
  • Submit

Company

  • Privacy Policy
  • Terms of Service
  • Refund Policy
  • Sitemap

Alternatives

  • Claude Code
  • Jira
  • Notion
  • Slack
  • Linear
  • Wispr Flow
  • All alternatives
© 2026 Open Source Alternatives. All rights reserved.
Open SourceAlternatives
Home/Categories/AI & Machine Learning/Ninfer
icon of Ninfer

Ninfer

  • 3.4K stars
  • Apache-2.0
  • Active this week
  • Last commit today

Last verified Nov 17, 2026

Run Qwen3 and GLM models locally via a native C++/CUDA inference engine with an OpenAI-compatible API, built for RTX 5090 and DGX Spark hardware.

Visit websiteGitHub3.4K starsDeploy on Hostinger
image of Ninfer
TL;DR

Ninfer is a native C++/CUDA inference engine for running Qwen3 and GLM models on RTX 5090 and DGX Spark hardware. It replaces paid inference APIs for teams running these specific GPU architectures, providing OpenAI- and Anthropic-compatible HTTP endpoints with no per-token charges. Licensed under Apache-2.0 and deployable from source or Docker. Best for ML engineers and AI infrastructure teams with on-premises NVIDIA hardware needing private, high-throughput local serving.

Read the full breakdownHide the full breakdownWho it is for, how it works, trade-offs, the quick start and the FAQ
who it's for

Who Ninfer is for#

ML engineers running Qwen3 models on RTX 5090

Ninfer provides the full serving stack: CLI for single requests, ninfer-serve for persistent HTTP service, and perplexity scoring. The OpenAI-compatible endpoint means existing Python clients using the openai library connect without changes. TOML configuration files manage model path, context limits, and speculative decoding settings.

Skip if:

You need to run models other than the documented Qwen3 and GLM checkpoints, or your hardware is a different GPU architecture. NInfer's model support is intentionally narrow.

AI infrastructure teams with data-residency requirements

Inference traffic stays entirely on-premises: no data leaves the machine. The OpenAI-compatible API means the team's existing tooling connects to a local endpoint instead of a cloud API, without application-layer changes. The Apache-2.0 license allows internal commercial use without licensing negotiations.

Skip if:

Your team lacks RTX 5090 or DGX Spark hardware, or you need a model not in NInfer's supported set. Cloud inference APIs are the practical alternative for other hardware.

Research teams evaluating Qwen3 or GLM model behavior

The ninfer-perplexity binary scores model outputs on custom text corpora. Capability evaluations in the README were run through NInfer's OpenAI-compatible route with EvalScope, so the same setup reproduces the published benchmark numbers. The ninfer binary writes answer content to stdout and scoring metadata to stderr, keeping them separable in pipelines.

Skip if:

Your evaluation infrastructure is built around Python-native inference (HuggingFace Transformers, vLLM). NInfer does not expose a Python API; interaction is via CLI arguments or HTTP.

the problem

The problem it solves#

Running large language models locally at high throughput is hard: Python-based inference stacks add framework overhead, and existing open source serving tools often target a broader set of hardware without tuning to specific GPU architectures. Teams using RTX 5090 or DGX Spark nodes that want to avoid paid inference APIs face a gap: either use a general-purpose server that leaves significant GPU performance on the table, or pay per-token to a commercial API that exposes inference traffic to a third party.

The harder part is speculative decoding and distributed multi-GPU inference: wiring these together efficiently requires deep integration with the CUDA execution model that many inference frameworks trade away for portability. For research labs and enterprise teams with data-residency requirements, neither the cloud API route nor a suboptimal local stack is acceptable.

how Ninfer solves it

How it solves it#

OpenAI- and Anthropic-compatible HTTP API

ninfer-serve exposes /v1/chat/completions and related endpoints, so existing OpenAI API clients connect without code changes. Anthropic-style endpoints are also supported. The server defaults to loopback for security; LAN access requires setting server.host = "0.0.0.0" and optionally a server.api_key in the TOML config.

Speculative decoding with MTP and DFlash2 backends

Speculative decoding runs with Multi-Token Prediction (MTP) and DFlash2 backends depending on the model and artifact. Measured MTP3 acceptance rates on RTX 5090 reach 46-69% depending on the model profile, boosting effective decode throughput at single-request concurrency.

CUDA Graph decode and shared-prefix caching

CUDA Graphs reduce kernel launch overhead for decode batches, contributing to the measured throughput. Shared-prefix caching retains exact text and multimodal prefixes, including the state needed to continue them, so repeated common prefixes (system prompts, long contexts) avoid redundant prefill computation.

Single-GPU and dual-DGX Spark distributed execution

The same codebase supports single RTX 5090 deployment and model-parallel execution across two DGX Spark nodes via TP2 and optional EP2. Dual-Spark Flash Next measured 109.5 to 811.9 tok/s steady decode at C=1 to C=32 concurrency in the November 2026 benchmark campaign.

Vision, tool calls, and constrained structured output

Every supported model profile handles text, Vision (image and video), tool calls, and constrained structured output. Vision and speculative decoding require the corresponding components in the downloaded artifact and must be enabled at startup via the TOML config.

FP8 and NVFP4 quantization formats

Artifacts are available in groupwise-int and NVFP4 quantization. FP8 KV cache is configurable at startup via --kv-dtype fp8. The Qwen3.6-27B NVFP4 profile measured 1,146.9 tok/s aggregate decode at C=8 on RTX 5090 in the documented benchmarks.

strengths · trade-offs

Strengths and trade-offs#

Strengths

  • Measured benchmark data for the specific hardware it targetsNinfer publishes its performance methodology and recorded measurements for RTX 5090 and DGX Spark, with configurations, engine versions, and cache formats. The benchmarks include concurrent decode saturation, single-request prefill throughput, and EvalScope capability evaluations. This is verifiable from the documented result pages rather than marketing claims.
  • Apache-2.0 license for unrestricted self-hosted useThe software license is Apache-2.0: you can run it on internal infrastructure, modify the source, and deploy it in commercial applications without licensing fees. Model weights have separate licenses documented in each model card, but the inference engine itself carries no commercial restrictions.
  • OpenAI-compatible serving with no per-token chargesOnce the model artifact is downloaded and the binary is running, inference requests carry no per-token cost. For high-volume private workloads, that is a fixed infrastructure cost versus recurring usage charges from paid inference services.
  • 90.4% GPQA-Diamond accuracy on Qwen3.8-27B NVFP4The Qwen3.8-27B NVFP4 profile scored 90.40% on GPQA-Diamond and 96.67% on AIME 2026 when evaluated through NInfer's OpenAI-compatible route with thinking enabled, MTP3, and EvalScope 1.9.0. These numbers confirm that the quantized artifacts served by NInfer preserve model capability.

Trade-offs

  • -Requires RTX 5090 or DGX Spark hardwareNInfer's supported deployments are specifically RTX 5090 (x86-64, sm_120a) and DGX Spark (AArch64, sm_121a). It does not run on consumer GPUs, cloud GPU instances, or older NVIDIA architectures. Teams without this specific hardware have no path to use it.
  • -Source build required; no prebuilt binariesInstallation requires building from source with CMake 3.28+, a C++20 host compiler, Ninja, CUDA 13.4, and several system libraries including FFmpeg and liburing. Docker simplifies the dependency chain for the RTX 5090 target, but there is no prebuilt binary download. First-time setup requires familiarity with CMake build systems.
  • -Limited model support; large weight downloads requiredNInfer supports only Qwen3.6-27B, Qwen3.8-27B, Qwen3.6-35B-A3B, Qwen3.8-Flash-Next, and GLM-5.3-Flash in the documented configurations. Flash Next artifacts total 101 GiB across four files; GLM totals 178 GiB across six. Custom weight conversion from other formats is documented but not automated.
  • -92 open issues as of November 2026The GitHub issue tracker shows 92 open issues (pull requests excluded, from GitHub's issue search API). For a project created in June 2026, that volume at 3,356 stars reflects active development and an evolving codebase. Expect rough edges, especially on edge cases in model configuration and distributed deployment.
versus alternatives

Ninfer vs alternatives#

NInfer vs Together AI

Together AI and similar paid inference services provide Qwen and other large language models via an API, charging per million tokens on managed cloud infrastructure. NInfer provides local inference for Qwen3 and GLM models on RTX 5090 or DGX Spark hardware, with no per-token charges after the hardware cost.

FeatureNInferTogether AI
LicenseApache-2.0Proprietary
Self-hostingYes (RTX 5090, DGX Spark)No
Per-token chargesNoneYes
Supported modelsQwen3, GLM (specific checkpoints)Qwen and others, managed
Data residencyOn-premisesThird-party cloud
OpenAI-compatible APIYesYes

NInfer is the better fit when you own RTX 5090 or DGX Spark hardware, need inference data to stay on-premises, or run high enough volume that per-token API charges exceed your infrastructure cost. The Qwen3.6-27B NVFP4 profile measured 1,146.9 tok/s aggregate decode at C=8 on RTX 5090.

Together AI and other paid inference APIs are the better fit when you do not own compatible hardware, need a wider selection of models, or want to avoid the build-from-source setup and operational overhead of a self-hosted inference stack.

install · quick start

Quick start#

bash
Deploy NInfer using Docker with the NVIDIA Container Toolkit installed on the host.
```bash
docker build --tag ninfer:local .
docker run --rm --gpus '"device=0"' --publish 8080:8080 --volume "$PWD/models:/models:ro" ninfer:local ninfer-serve /models/qwen3_8_27b_nvfp4.ninfer --host 0.0.0.0 --max-context 8192 --kv-dtype fp8
```
tech stack · detected from GitHub

What it's built on#

Languages
C++Python
Frameworks
Vue
frequently asked

FAQ#

What hardware does NInfer require?

NInfer runs on RTX 5090 (x86-64, sm_120a) and DGX Spark (AArch64, sm_121a). These are the only supported targets. The build system checks the requested architecture at CMake time, and there is no general CUDA fallback for other NVIDIA GPUs.

How do I install NInfer?

The primary path is building from source on Linux with CMake 3.28+, a C++20 compiler, Ninja, CUDA 13.4, and system dependencies including FFmpeg, liburing, hwloc, and libjpeg. The repository includes a Dockerfile for the RTX 5090 target that handles the dependency chain. Run docker build --tag ninfer:local . then docker run with --gpus to start the server. No prebuilt binary downloads are available.

Which models does NInfer support?

NInfer supports Qwen3.6-27B, Qwen3.8-27B, Qwen3.6-35B-A3B, Qwen3.8-Flash-Next, and GLM-5.3-Flash. Official .ninfer artifacts are available on HuggingFace under the neroued organization. Flash Next and GLM have multi-file downloads (101 GiB and 178 GiB respectively). Custom weights require the weight conversion process documented in the repository.

Does NInfer support OpenAI-compatible clients?

Yes. ninfer-serve exposes /v1/chat/completions and other OpenAI-compatible endpoints. Any client library that speaks the OpenAI API format, including the openai Python package, can point to the local NInfer server. Anthropic-style endpoints are also supported. The server defaults to loopback (127.0.0.1:8080); set server.host = "0.0.0.0" in the TOML config for LAN access.

What is NInfer's license, and can I use it commercially?

NInfer is Apache-2.0 licensed: you can run it, modify it, and use it for internal commercial applications without paying licensing fees or seeking permission. Model weights are licensed separately: 27B and 35B-A3B artifacts are Apache-2.0; Flash Next combines the Qwen Community License 1.0 and NVIDIA Open Model License; GLM uses MIT. Check each model card before deploying a specific artifact in a commercial context.

also worth a look

Similar open-source tools#

Hermes Agent

Featured

Alternative toGrok Bot

Self-hosted AI agent: persistent memory, multi-channel chat, model choice.

  • 263,254 GitHub stars
  • MITLicence MIT

AutoRAG

Alternative toRagie AI

Local-first AI librarian for files, messengers, and cloud drives

  • 5,899 GitHub stars
  • Apache-2.0Licence Apache-2.0

SkillEvaluator

InAI & Machine Learning

Validation, deduplication, and live sandbox eval for AI agent skills.

  • 1,698 GitHub stars
  • Apache-2.0Licence Apache-2.0

Llama.cpp

Alternative toOpenAI API

C/C++ LLM inference engine with OpenAI-compatible API, no cloud.

  • 135,212 GitHub stars
  • MITLicence MIT

HA-MCP

Alternative toJosh.ai

Natural language Home Assistant control via MCP, MIT licensed.

  • 7,757 GitHub stars
  • MITLicence MIT

Elephant Goldfish Model

Alternative toAmazon Kiro

Draft specs with context, review with fresh zero-context agents.

  • 19,835 GitHub stars
  • Apache-2.0Licence Apache-2.0
by category

Open source tools in these categories#

543

AI & Machine Learning

freeCodeCamp, REA (Reverse Engineer Anything), OpenClaw
140

LLMOps & AI Tooling

Hermes Agent, OpenCode, MarkItDown
660

Developer Tools

REA (Reverse Engineer Anything), Skills for Real Engineers, superpowers
Hostinger

Self-host Ninfer on a Hostinger VPS

Keep it running 24/7, even when your laptop is off. A Hostinger VPS gives it a server of its own.

  1. 01

    Choose a plan

    Starts from $6.49/mo

  2. 02

    Apply

    Extra 10% off yearly plans

  3. 03

    Install Ninfer

    On a server that stays on

  4. 04

    Start building

    Focus on the app, not the server

Deploy on Hostinger

Sponsors

Inbox Zero

Alternative toSuperhuman

AI email assistant that drafts replies and clears your inbox

  • 13,025 GitHub stars
  • AGPL-3.0Licence AGPL-3.0
  • #Email

rakazo

Alternative toGrok Bot

AI teammates you own: your keys, your model, your machine.

  • 8,291 GitHub stars
  • Apache-2.0Licence Apache-2.0
  • #AI Agents

SnitchFeed

Live monitoring on LinkedIn, X, Reddit & more. Agentic social listening for startups.

Marketing & Growth

Advertise your product

New open source projects by email

The projects we add and every new guide. Free, unsubscribe any time.

Find open source alternatives to paid software

/
  • Claude Code to OpenCode
  • Jira to Windshift
  • Notion to AppFlowy
  • Slack to Mattermost
  • Linear to GitLab
  • Wispr Flow to Handy
  • Arc to Zen Browser
  • CapCut to Concat
  • Grok Bot to rakazo
  • Superhuman to Inbox Zero
  • Discord to Stoat
  • Airtable to AITable
  • Datadog to SigNoz
  • Muse to OpenClaw
  • Railway to Coolify
  • GitHub Copilot to Cline
  • Confluence to Wiki.js
  • Microsoft Teams to Rocket.Chat
  • 1Password to KeePassXC
  • Firebase to Supabase
  • Pinecone to Qdrant
  • Zapier to n8n
  • Devin AI to OpenHands
  • Asana to Leantime
  • Claude Code to OpenCode
  • Jira to Windshift
  • Notion to AppFlowy
  • Slack to Mattermost
  • Linear to GitLab
  • Wispr Flow to Handy
  • Arc to Zen Browser
  • CapCut to Concat
  • Grok Bot to rakazo
  • Superhuman to Inbox Zero
  • Discord to Stoat
  • Airtable to AITable
  • Datadog to SigNoz
  • Muse to OpenClaw
  • Railway to Coolify
  • GitHub Copilot to Cline
  • Confluence to Wiki.js
  • Microsoft Teams to Rocket.Chat
  • 1Password to KeePassXC
  • Firebase to Supabase
  • Pinecone to Qdrant
  • Zapier to n8n
  • Devin AI to OpenHands
  • Asana to Leantime