Who llmfit is for#
ML engineers choosing models for local inference
llmfit ranks every model in its catalog against your workstation's VRAM and RAM before you download anything. The ranked table covers fit, speed, and context length in one view, replacing the trial-and-error cycle of loading and rejecting models one at a time.
Skip if:
You are running inference on a cloud GPU cluster with fixed hardware. llmfit adds no value when you are not selecting a model for hardware you own.
Developers comparing quantization levels
Dynamic quantization selection lets you see how different quant levels affect memory fit and estimated speed for the same base model on your hardware. The `llmfit info` command shows the estimate basis for any model-quant combination before you commit to a download.
Skip if:
You already know the exact model and quantization you need and just want to run it. llmfit is most useful in the selection phase, not the serving phase.
Teams contributing shared hardware benchmark data
The built-in bench flow measures real tok/s and time-to-first-token, saves results locally, and lets you submit them back to the project as a pull request from the TUI. Each merged submission improves estimate accuracy for anyone running identical hardware.
Skip if:
Your organization prohibits contributing usage data or submitting pull requests to public repositories. Submission is fully opt-in; results are always saved locally first.
The problem it solves#
Running local LLMs on personal or workstation hardware is a trial-and-error process. A model that fits in VRAM on paper may swap to system RAM at runtime, dropping inference speed from tens of tokens per second to single digits. MoE architectures like DeepSeek-V3 complicate estimates further: naive memory calculations use total parameter count rather than the active subset, overestimating RAM requirements by a wide margin.
Most developers resort to reading community posts, checking model cards, or downloading and running models one at a time. None of those approaches account for the interaction between your specific CPU, GPU, VRAM, and system RAM. The result is wasted time, failed loads, and wasted storage before any actual inference work begins.
How it solves it#
Hardware detection and model scoring
llmfit reads your RAM, CPU, GPU, and VRAM on launch and scores every model in its catalog across four dimensions: memory fit, estimated speed, quality, and context length. The score reflects your actual hardware configuration, not a generic recommendation.
Speed estimation from memory-bandwidth model
Token-per-second estimates come from a memory-bandwidth formula grounded in runtime sampling and real community benchmarks. The `llmfit info` command shows the exact inputs for each estimate and the commands to verify them on your machine.
Interactive TUI and scriptable CLI
The default interactive TUI shows your detected hardware at the top and every model ranked in real time. CLI mode supports scripts and automation pipelines: `llmfit fit` prints a ranked table, and `llmfit recommend --json` returns top picks as JSON for agent or script consumption.
MoE architecture support
llmfit understands Mixture-of-Experts architectures and estimates memory from the active parameter subset rather than total parameter count. This produces accurate fit results for models like Mixtral and DeepSeek-V3, which standard memory calculators overestimate significantly.
Community benchmark contribution
A built-in benchmarking flow lets you measure real tok/s and time-to-first-token against your running local provider, then submit results back to the project as a pull request from the TUI. No GitHub CLI or third-party account required; results are saved locally before any submission.
Multiple local runtime providers
Supports Ollama, llama.cpp, MLX, Docker Model Runner, and LM Studio as inference backends. The `llmfit doctor` command generates a hardware detection report for debugging provider integration issues.
Strengths and trade-offs#
Strengths
- MoE-aware memory estimationStandard model memory calculators treat every model as dense, overestimating RAM requirements for Mixture-of-Experts models like Mixtral and DeepSeek-V3 by a large factor. llmfit estimates from the active parameter subset, so fit scores reflect what the model actually loads into memory.
- Transparent, verifiable speed estimatesEvery speed estimate ships with its inputs: the memory-bandwidth assumptions, the runtime sampling basis, and the community measurements that ground it. The `llmfit info` command shows exactly what each number assumes, and each estimate includes commands to verify results on your own hardware.
- MIT license with no server dependencyllmfit is MIT licensed and runs as a single local binary with no database, no API calls on launch, and no account required. It contacts external services only when you explicitly trigger model downloads, provider queries, or the community leaderboard.
Trade-offs
- -Speed numbers are estimates, not guaranteesPerformance estimates are based on a memory-bandwidth model and community benchmarks, not live tests from your exact hardware. Real throughput can differ, especially on unusual GPU or CPU combinations or for quantizations not yet in the community dataset. The `llmfit bench` command lets you replace estimates with measured values, but it requires a running model and provider first.
- -Catalog coverage depends on community contributionsModels not yet in llmfit's catalog will not appear in fit results. Very new releases or obscure fine-tunes may be missing until someone adds them. The custom models guide covers adding local-only entries without a rebuild, but the built-in catalog is community-maintained and grows through contributions.
llmfit vs alternatives#
llmfit vs NVIDIA AI Enterprise
NVIDIA AI Enterprise is an enterprise platform for deploying, managing, and scaling AI workloads on NVIDIA GPU infrastructure. It targets organizations running models in production at scale, with support for multi-node clusters, enterprise security, and managed inference microservices.
| Feature | llmfit | NVIDIA AI Enterprise |
|---|---|---|
| License | MIT | Proprietary |
| Deployment | Local binary, no server | Managed on NVIDIA hardware |
| Target scale | Single machine or workstation | Multi-node GPU clusters |
| Cost | Free | Per-GPU enterprise licensing |
| Hardware scope | Any hardware you own | NVIDIA GPUs only |
llmfit is the better choice when your goal is to understand what runs on hardware you already own, select the right model before downloading it, or build a local inference stack without a GPU fleet. NVIDIA AI Enterprise is the appropriate choice for production deployments on NVIDIA GPU infrastructure, managed model serving at scale, and enterprise compliance requirements that a single-binary CLI cannot address.
llmfit vs OctoAI
OctoAI is a managed cloud inference platform that runs optimized model endpoints on behalf of the caller. You call an API; OctoAI handles GPU allocation, model loading, and scaling. It is designed for teams that want fast inference without managing hardware.
| Feature | llmfit | OctoAI |
|---|---|---|
| License | MIT | Proprietary |
| Deployment | Local, no server required | Managed cloud API |
| Hardware | Your own machine | None (cloud) |
| Cost | Free | Per-token API pricing |
| Offline capable | Yes | No |
llmfit makes sense when you run models locally, want to avoid per-token API costs, or need offline inference. OctoAI is better when you need production-grade latency guarantees, want to avoid managing hardware entirely, or your workload exceeds what a single machine can serve. The two tools address different parts of the LLM stack: llmfit helps you select and fit models for your own hardware, while OctoAI removes the hardware question by hosting inference in the cloud.
Install and self-host#
Install llmfit via Homebrew on macOS/Linux, pip via uv, or Docker for a containerized run.
```bash
brew install llmfit
uv tool install -U llmfit
docker run ghcr.io/alexsjones/llmfit
```What it's built on#
- Languages
- JavaScriptPythonRust
- Frameworks
- React
FAQ#
Does llmfit download models, or does it only recommend them?
llmfit does not download models. It detects your hardware, scores models from its catalog, and tells you which ones fit and how fast they will run. Downloading and serving happen through your local runtime provider, such as Ollama or llama.cpp. The bench command measures performance after you have already loaded a model through your provider.
Is llmfit free to use?
Yes. llmfit is MIT licensed and free to install and run with no account, API key, or paid tier. It contacts external services only when you explicitly trigger model downloads, provider queries, or the community leaderboard.
How accurate are llmfit's speed estimates?
Speed estimates use a memory-bandwidth model grounded in runtime sampling and community benchmarks. They are accurate enough for useful model comparisons on the same hardware, but not a substitute for measured results. The llmfit bench command replaces estimates with real tok/s numbers from your running provider on your actual hardware.
Does llmfit support Apple Silicon and MLX?
Yes. MLX is one of the supported local runtime providers. llmfit handles Apple Silicon GPU detection and uses MLX as a backend option for model scoring and serving. The platform support documentation in the repository covers macOS-specific detection details.
Can I add models that are not in llmfit's built-in catalog?
Yes. The custom models guide in the repository covers adding models locally without a rebuild. You can also contribute new models to the built-in catalog via a pull request; the catalog is community-maintained and grows through contributions from users with new hardware and model combinations.
Similar open-source tools#
OpenCode
OpenCode is an open-source AI coding agent that assists developers in
omlx
Fast LLM inference for Apple Silicon with persistent SSD KV caching
ai-memory
Persist AI agent context across sessions and tool switches
Soup
Fine-tune any LLM on a 4 GB GPU, one YAML config
Switchyard
LLM proxy with API translation and multi-backend routing
DwarfStar (ds4)
Local inference engine for DeepSeek V4 Flash/PRO and GLM 5.2