TensorFold

1.6K stars
Apache-2.0
Active this week
Last commit today

Last verified Nov 1, 2026

Run open language models on Apple Silicon and NVIDIA GPUs from a Zig binary with native Metal and CUDA kernels, no Python required.

Visit website GitHub1.6K Deploy on Hostinger

Read the full breakdownWho it is for, how it works, trade-offs, the quick start and the FAQ

who it's for

Who TensorFold is for#

ML engineers serving open models on Apple Silicon

TensorFold is the lowest-friction path to an OpenAI-compatible endpoint on an M-series Mac. One Homebrew install, one pull command, and one serve command gets a quantized model running locally. Engineers building coding assistants, RAG pipelines, or agentic workflows get GPU-accelerated inference without Python, Docker, or a separate server process.

Skip if:

Teams that need inference on hardware without Metal or CUDA support, or on Linux without a qualifying NVIDIA driver (R580 or newer). TensorFold has no CPU-only inference backend.

Teams replacing cloud API costs with hardware they own

High-volume inference workloads face recurring per-token costs from commercial providers. TensorFold turns GPU hardware the team already owns into a local inference service with no per-token fees. The OpenAI-compatible API means existing applications change the endpoint URL and run unchanged, with no client-side migration.

Skip if:

Workloads that require models larger than available local GPU memory, or teams without qualifying Apple Silicon or NVIDIA hardware. Those cases still need a cloud option for the models that do not fit.

Developers building and testing AI apps on macOS

The TensorFold desktop app bundles the engine, model downloads, live generation metrics, and coding workers into an Apple Silicon app. Developers test and iterate on local model calls without configuring a separate server process. The app documents hardware recommendations per model and includes document editing with a local LLM.

Skip if:

Linux-first or Windows-first developers. The desktop app is macOS-only; Linux and Windows use the CLI, and Windows requires a native build rather than a packaged binary.

Researchers benchmarking quantization formats on specific hardware

The project maintains a community leaderboard with receipts: token hashes, hardware names, and per-prompt verification for each contributed benchmark entry. Researchers can run a qualified checkpoint, compare against recorded runs, and submit their own hardware reading. The model table in the README documents which quantization formats are natively served and which require conversion first.

Skip if:

Researchers who need CPU-only or platform-agnostic reproducibility. TensorFold's model table and leaderboard are scoped to Apple Silicon and NVIDIA hardware.

the problem

The problem it solves#

Running large language models in production means paying per token to cloud API providers, sending data off your servers, and accepting the latency of a remote inference round trip. For high-volume workloads, those per-token costs accumulate into recurring monthly bills, while the GPU hardware that would serve the same requests is a one-time expense.

The technical challenge for local inference has always been setup complexity. Most local inference approaches depend on Python, CUDA libraries, and model conversion pipelines that take hours to configure and break across OS and driver versions. Developers end up choosing between cloud API costs and a fragile local stack, neither of which scales well without dedicated infrastructure work.

how TensorFold solves it

How it solves it#

Zero Python dependency

TensorFold serves models from a single Zig binary that reads checkpoints, tokenizes requests, and runs Metal or CUDA kernels directly. No Python installation, no MLX, no virtual environments to manage. The binary is self-contained: the Homebrew tap on macOS covers the engine and all its dependencies in one install command.

OpenAI and Anthropic API wire compatibility

The server exposes /v1/chat/completions, completions, Responses, and Anthropic Messages endpoints, so any client already calling the OpenAI or Anthropic APIs works without code changes. Point it at http://127.0.0.1:8080 and the application sees a local model as a drop-in replacement. The /v1/models endpoint lists the served model name so existing model-selection logic works unchanged.

Speculative decoding with drafter models

On supported models, a smaller drafter proposes several tokens and the target model verifies them together, committing only the matching prefix. This raises throughput on a single stream: Nemotron 3.5 Lightning reached 874 tokens per second on an RTX PRO 6000 Blackwell with drafting enabled, as documented in the project's release benchmarks.

Prompt reuse across requests

The engine retains already-computed work across requests. When a new request shares a prefix with a prior one, TensorFold skips recomputing those tokens. The project records prompt reuse latency at 90 to 110 ms for Nemotron on an NVIDIA GB10, which matters for chat applications with long, repeated system prompts.

Multi-machine model sharding

Models that exceed a single machine's memory can split across two Apple Silicon Macs connected via Thunderbolt 5. GLM-5.3-Flash is documented for two M5 Ultras running at 199 tokens per second aggregate across both machines. The engine routes the forward pass across the nodes; each machine holds its shard of the weights.

Living Weights in-checkpoint learning

A --slide flag teaches a running Nemotron 3.5 Lightning server new facts by writing them into the checkpoint files. Feed it a document and the engine extracts and learns facts one by one. The feature is experimental: the README documents that recall drops past a few dozen facts, related facts can disturb each other, and learning costs some decode speed on Mac.

strengths · trade-offs

Strengths and trade-offs#

Strengths

No Python in the serving pathThe binary runs Metal and CUDA kernels directly, so macOS system updates and NVIDIA driver upgrades do not break a Python environment. On macOS, Homebrew manages the installation and its dependencies; there is no pip, no virtual environment, and no requirements.txt. The README explicitly states that serving needs no Python or MLX installation.
Apache-2.0 license permits commercial useApache-2.0 allows running TensorFold in commercial products and services, including building hosted inference on top of it, without licensing fees or copyleft obligations. Teams can distribute modified binaries and run the engine as part of a paid service; keeping the license notice in distributed copies is the only requirement.
Drop-in API replacement for existing OpenAI and Anthropic clientsApps built against the OpenAI or Anthropic wire formats can move to local inference by changing the base URL and nothing else. No client-side code changes, no SDK swap, no migration effort on the caller. The server speaks both formats from the same process, so a single TensorFold instance can serve clients from both API families.
Auditable hardware-specific benchmarksEvery performance number the project publishes names the specific model, hardware, release, and decode conditions. The community leaderboard links to receipts with token hashes and per-prompt verification for contributed runs. Numbers tied to named hardware and a specific release version are more useful to capacity planners than throughput figures from unspecified conditions.

Trade-offs

-CUDA engine requires NVIDIA driver R580 or newerThe CUDA engine is built with CUDA 13.0 and refuses an older driver at startup. Teams on NVIDIA hardware with drivers below R580 need to upgrade before TensorFold will run on those machines. The Linux installation guide notes this requirement and directs users to check the installed driver version with nvidia-smi.
-Linux install requires manual archive extractionThere is no Linux package manager path: you download a versioned release archive, verify its SHA-256 checksum, extract it, add the bin/ directory to your PATH, and set a shell function alias. The macOS Homebrew install is one command; the Linux path is four to six steps that server deployments need to script or automate.
-110 open issues on a project created in June 2026TensorFold launched in June 2026 and carries 110 open issues. For a project this young, some rough edges in the supported model matrix and platform coverage are under active work. Verify that your specific model, quantization format, and hardware appear in the qualified table in the README before committing to a production workload.
-Living Weights has documented recall limitationsThe in-checkpoint learning feature is explicitly experimental. The README states that recall drops past a few dozen learned facts, related facts can disturb each other, and learned facts cost decode speed on Mac. It should not be used where recall reliability matters. The README documents tensorfold slide rollback to undo lessons and restore the original checkpoint.

versus alternatives

TensorFold vs alternatives#

TensorFold vs Cloud LLM APIs (OpenAI, Anthropic)

Local inference with TensorFold and cloud API access from providers like OpenAI and Anthropic both deliver a model over an HTTP API. The difference is cost structure, data locality, and hardware control.

Feature	TensorFold	Cloud LLM APIs
License	Apache-2.0	Proprietary
Self-hosting	Yes	No
Cost model	One-time hardware cost, no per-token fees	Per-token metered billing
Data locality	Stays on your infrastructure	Sent to provider
API format	OpenAI and Anthropic wire format	Native
Model selection	Open models from Hugging Face	Provider's hosted models

TensorFold is the better fit when you are running inference at a scale where per-token costs are significant, processing data that cannot leave your infrastructure, or when hardware you already own can offset recurring API spend. Teams with Apple Silicon Macs or NVIDIA GPUs can run 2B to 30B+ parameter models locally with no monthly billing and no data leaving their network.

Cloud LLM APIs remain the better choice when you need frontier-model capability that open models do not yet match, when you have no qualifying hardware, or when you need to absorb inference spikes without provisioning more machines. For teams without M-series Macs or recent NVIDIA GPUs, a metered cloud API is simpler to operate than a self-hosted inference engine.

install · quick start

Quick start#

bash

Install TensorFold on Apple Silicon via Homebrew, then pull a model and start serving it.
```bash
brew install ashhart/tensorfold/tensorfold
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking
```

tech stack · detected from GitHub

What it's built on#

Languages: JavaScriptPythonTypeScriptZig
Frameworks: ElectronReact

frequently asked

FAQ#

Does TensorFold require Python or any Python packages to serve models?

No. The engine is a single Zig binary that runs Metal and CUDA kernels directly without Python, MLX, or virtual environments. On macOS, install it via Homebrew; on Linux, download the release archive and add the bin directory to your PATH. The README states explicitly that serving needs no Python or MLX installation.

Which GPUs and operating systems does TensorFold support?

TensorFold supports Apple Silicon Macs (M1 through M5, macOS 13.3 or later) via Metal, and NVIDIA GPUs via CUDA 13.0 (driver R580 or newer required). Linux builds cover x86_64 and aarch64 hosts. Windows support requires a native build; a guide is in the project docs. The repo description also lists Vulkan as a supported backend.

Is TensorFold free to use in commercial products?

Yes. TensorFold is Apache-2.0 licensed, which permits commercial use, modification, and redistribution without licensing fees. You can run it as part of a paid service or build commercial products on top of it. The only requirement is keeping the license notice in distributed copies.

Which language models can TensorFold serve?

The README documents tested models including Qwen3.5-2B and Qwen3.8-27B from Alibaba, NVIDIA Nemotron 3.5 Lightning, GLM-5.3-Flash from Z.ai, Gemma 4, LFM2.5 from Liquid AI, and others. Most are pulled via tensorfold pull from Hugging Face repositories; some require conversion from upstream weights first. Check the model table in the GitHub README for hardware requirements and native versus conversion status per checkpoint.

Can TensorFold serve models that require more memory than a single machine holds?

Yes, for models with a documented multi-machine recipe. GLM-5.3-Flash is qualified for two M5 Ultra Macs connected via Thunderbolt 5 using the Connected Macs feature. Each machine holds its shard of the weights and the engine routes the forward pass across both. The model table in the README lists which models have multi-machine support and the required hardware configuration.

also worth a look

Similar open-source tools#

Hermes Agent

Featured

Alternative toGrok Bot

Self-hosted AI agent: persistent memory, multi-channel chat, model choice.

258,472 GitHub stars
Licence MIT

Mlx Serve

Alternative toLM Studio

Local LLM server for Apple Silicon, faster than LM Studio

2,206 GitHub stars
Licence MIT

Skillshare

InDeveloper Tools

AI coding setup manager for skills, agents, MCP, and hooks

2,872 GitHub stars
Licence MIT

Agent Desktop

InAI & Machine Learning

Desktop computer use for AI agents via OS accessibility tree

3,850 GitHub stars
Licence Apache-2.0

Aletheia Mem

InDeveloper Tools

Persistent local memory and rules for Claude Code, Codex and Cursor

9 GitHub stars
Licence Apache-2.0

Claude Cookbooks

InAI & Machine Learning

Official Claude API notebooks and code recipes for Python developers

53,978 GitHub stars
Licence MIT

by category

Open source tools in these categories#

446

Self-host TensorFold on a Hostinger VPS

Keep it running 24/7, even when your laptop is off. A Hostinger VPS gives it a server of its own.

01
Choose a plan
Starts from $6.49/mo
02
Apply
Extra 10% off yearly plans
03
Install TensorFold
On a server that stays on
04
Start building
Focus on the app, not the server

Deploy on Hostinger

Inbox Zero

Alternative toSuperhuman

AI email assistant that drafts replies and clears your inbox

12,855 GitHub stars
Licence AGPL-3.0
#Email

rakazo

Alternative toGrok Bot

AI teammates you own: your keys, your model, your machine.

6,242 GitHub stars
Licence Apache-2.0
#AI Agents

SnitchFeed

Live monitoring on LinkedIn, X, Reddit & more. Agentic social listening for startups.

Marketing & Growth

Advertise your product

The projects we add and every new guide. Free, unsubscribe any time.

Find open source alternatives to paid software

TensorFold vs Cloud LLM APIs (OpenAI, Anthropic)

Feature	TensorFold	Cloud LLM APIs
License	Apache-2.0	Proprietary
Self-hosting	Yes	No
Cost model	One-time hardware cost, no per-token fees	Per-token metered billing
Data locality	Stays on your infrastructure	Sent to provider
API format	OpenAI and Anthropic wire format	Native
Model selection	Open models from Hugging Face	Provider's hosted models

Who TensorFold is for#

ML engineers serving open models on Apple Silicon

Skip if:

Teams replacing cloud API costs with hardware they own

Skip if:

Developers building and testing AI apps on macOS

Skip if:

Researchers benchmarking quantization formats on specific hardware

Skip if:

The problem it solves#

How it solves it#

Zero Python dependency

OpenAI and Anthropic API wire compatibility

Speculative decoding with drafter models

Prompt reuse across requests

Multi-machine model sharding

Living Weights in-checkpoint learning

Strengths and trade-offs#

Strengths

Trade-offs

TensorFold vs alternatives#

TensorFold vs Cloud LLM APIs (OpenAI, Anthropic)

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

LLMOps & AI Tooling

Developer Tools

Find open source alternatives to paid software

Who TensorFold is for#

ML engineers serving open models on Apple Silicon

Skip if:

Teams replacing cloud API costs with hardware they own

Skip if:

Developers building and testing AI apps on macOS

Skip if:

Researchers benchmarking quantization formats on specific hardware

Skip if:

The problem it solves#

How it solves it#

Zero Python dependency

OpenAI and Anthropic API wire compatibility

Speculative decoding with drafter models

Prompt reuse across requests

Multi-machine model sharding

Living Weights in-checkpoint learning

Strengths and trade-offs#

Strengths

Trade-offs

TensorFold vs alternatives#

TensorFold vs Cloud LLM APIs (OpenAI, Anthropic)

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

LLMOps & AI Tooling

Developer Tools

Find open source alternatives to paid software