Who Soup is for#
ML engineers fine-tuning on local consumer hardware
Soup's layer streaming lets a single 4 GB GPU run fine-tuning jobs that previously required renting cloud compute. The YAML-first workflow and auto-detection handle most config decisions, so engineers spend time on data and evaluation rather than tuning training parameters.
Skip if:
Your team has dedicated high-VRAM GPUs (24 GB+) where a resident run is straightforward, or your organization already has managed training infrastructure. Layer streaming's throughput penalty is not worth taking on capable hardware.
Researchers building preference-trained or RLHF models
DPO, ORPO, SimPO, KTO, GRPO, and PPO all run through the same YAML interface. The `soup advise` command ranks training methods against your data before you commit GPU hours. `soup data doctor` runs 8 chat-template checks that catch the EOS bug and other silent data failures before training starts.
Skip if:
Your preference training requires online DPO against a remote judge model that cannot run locally. GRPO and PPO with online reward models have specific infrastructure requirements not covered by the consumer-GPU path.
Teams migrating from LLaMA-Factory, Axolotl, or Unsloth
`soup migrate` converts existing configs in one command and preserves training hyperparameters. Teams can try Soup's `soup ship` release gate and semantic data dedup without rewriting their entire pipeline first.
Skip if:
Your existing config uses advanced features like custom data pipelines, non-standard PEFT variants, or multi-node setups that may not convert cleanly. Verify the output config against your expected training behavior before committing to the migration.
Organizations that need compliance-ready ML pipelines
`soup init --template hipaa` (or soc2, eu-ai-act, sr-11-7) starts from a regulation-shaped config. `soup bom` and `soup attest` emit signed ML-BOMs and SLSA-3 provenance. `soup ci init` blocks merges on DON'T SHIP verdicts. For teams with audit requirements, this is a self-hosted path to documented, reproducible, signed training runs.
Skip if:
Your compliance requirements mandate a certified managed service with SLA guarantees. Self-hosted tooling satisfies technical controls but does not substitute for vendor certification in all frameworks.
The problem it solves#
LLM fine-tuning demands either significant cloud spend or deep infrastructure knowledge. Managed training services from providers like Together AI, Hugging Face AutoTrain, and Google Cloud Vertex AI charge per GPU-hour, require uploading your data to their servers, and lock your workflow into proprietary tooling. Rolling your own training stack means wiring together PyTorch, PEFT, TRL, quantization libraries, and experiment tracking by hand, with no pre-flight checks to catch silent failures before they waste GPU hours.
The deeper problem is hardware: most developers and researchers work on consumer GPUs with 4 to 8 GB of VRAM, which is not enough to load a 7 to 8 billion parameter model resident in memory, let alone train one. That ceiling has historically meant renting cloud GPUs for every experiment, even quick iteration runs that do not justify the cost.
How it solves it#
Layer Streaming (BETA)
Streams the frozen base model from CPU RAM or NVMe one decoder layer at a time, quantizing to NF4 on the fly. Measured result: Llama-3.1-8B trains at 119.6 tok/s in 3.32 GB peak VRAM on a 4 GB RTX 3050 Laptop. DPO, ORPO, SimPO, and KTO stream too; the DPO reference model reuses the same streamed base at 0.914x the supervised peak, adding no extra weights. Currently BETA and limited to transformers, text, and plain LoRA.
23 Training Methods, 142 Ready Recipes
Supports SFT, DPO, ORPO, SimPO, KTO, GRPO, PPO, IPO, BCO, distillation, classification, vision, audio, and more across 23 methods. 142 pre-built recipes cover major model families so most tasks start from a tested baseline rather than a blank config. `soup recipes list` shows the full catalog.
Automatic Config Generation
`soup init` runs an interactive wizard or starts from a named template (chat, code, tool-calling, medical, reasoning, vision, and 15 more). Soup auto-detects batch size, target modules, optimizer, and scheduler from the task and hardware, eliminating the most common causes of misconfigured training runs.
`soup ship`: Release Gate for Weights
Runs the fine-tuned model against seven offline suites (MCQ, arithmetic, tool-calling, JSON validity, safety/refusal) and returns a binary SHIP or DON'T SHIP verdict with committable evidence. `--noise-floor N` re-runs the base model N times to measure GPU non-determinism before calling a delta significant. Exit codes: 0 = SHIP, 2 = DON'T SHIP, 1 = runtime error.
Migration from LLaMA-Factory, Axolotl, and Unsloth
`soup migrate --from llamafactory config.yaml` converts an existing config to soup.yaml format automatically. Supports LLaMA-Factory, Axolotl, and Unsloth notebook conversion. No manual rewriting; the converter handles field mapping and adds Soup's auto-detected parameters for what it cannot infer.
MCP Server for Coding Agents
`soup mcp serve` exposes Soup to any MCP client (Claude Code, Cursor, Cline, Continue) over stdio. 14 read-only tools return JSON; 2 plan-only mutating tools allow training without a terminal. No network listener required.
Strengths and trade-offs#
Strengths
- Trains 8B Models on a 4 GB Consumer GPUThe layer-streaming BETA is the only open source training path that can fit Llama-3.1-8B on a card with 4 GB of VRAM, measured on a real RTX 3050 Laptop and independently verified on a borrowed 8x H100. For developers without a cloud budget or a high-VRAM GPU, this is the enabling capability. Comparable cloud training runs on Together AI or Vertex AI cost money per GPU-hour and send data off-premise.
- Honest, Disclosed BenchmarksEvery measurement on the Soup website and README identifies the hardware, the version, and what changed since. The v0.73.0 release notes document a silent gradient bug at 32B+ NF4 (fixed), a prior adapter-key bug that made adapters load as a no-op (fixed in v0.72.1), and three withdrawn readings. This level of disclosure is uncommon in the space and makes performance numbers trustworthy.
- Apache-2.0 License with No Usage RestrictionsApache-2.0 allows commercial use, modification, and distribution without source-publishing obligations. There is no enterprise tier required for production use, no API key tied to training runs, and no telemetry. The source is on GitHub under the same license used for deployment.
- Integrates Across the ML StackFirst-class support for Unsloth (2-5x faster training), Apple MLX (M1 to M4 native), DeepSpeed (ZeRO 2/3/++), FlashAttention v2/v3, vLLM, SGLang, Ollama, llama.cpp, HuggingFace Hub push, and W&B/MLflow/SwanLab experiment tracking. Models train on Soup and deploy with the same tools the team already uses.
Trade-offs
- -Layer Streaming Is Still BETALayer streaming is limited to transformers, text, and plain LoRA. The reference model costs no extra memory but reads the layer stack 1.52x as often per step, slowing training versus a resident run. The 32B+ NF4 backward-pass bug (fixed in v0.73.0) and the v0.72.0 adapter-key silent no-op bug (fixed in v0.72.1) show that edge cases in this path are still being found. No tok/s is claimed for preference losses on 4 GB hardware because none was measured.
- -Python 3.10 to 3.12 OnlyPython 3.13 is not supported. On 3.13+, pip resolves untested PyTorch wheels that crash in the native extension before Soup starts. If your environment is pinned to a newer Python version, you need a virtual environment or container on a supported version before installing.
- -Self-Hosting Requires a GPU for TrainingCPU training is experimental and very slow per the README. Apple Silicon (MPS) works but requires the `[mlx]` extra. Production-quality fine-tuning requires a CUDA GPU. If you have no GPU access and cannot use cloud credits, Soup's value is limited to data tooling and model serving rather than training.
Soup vs alternatives#
Soup vs Together AI
Both Soup and Together AI offer fine-tuning for LLMs including Llama and Qwen family models, but they operate on opposite sides of the build-vs-buy line. Together AI is a managed cloud API; Soup is a self-hosted CLI.
| Feature | Soup | Together AI |
|---|---|---|
| License | Apache-2.0 | Proprietary |
| Self-hosting | Yes, required | No |
| Runs on 4 GB consumer GPU | Yes (layer streaming BETA) | No |
| Data leaves your machine | No | Yes |
| Pricing | Free (your hardware) | Per-GPU-hour |
| Training methods | 23 methods, 142 recipes | Managed selection |
Soup is the stronger choice when your training data is sensitive, your budget is limited, or you need to iterate quickly on a consumer GPU. Together AI is worth considering when you need zero infrastructure management, access to high-VRAM clusters for models above 70B, or managed multi-node training without configuring DeepSpeed yourself.
Soup vs Hugging Face AutoTrain
Hugging Face AutoTrain is a managed fine-tuning service on the HuggingFace platform. Like Soup, it supports LoRA fine-tuning for popular model families. Unlike Soup, it runs on Hugging Face's cloud infrastructure, requires uploading your dataset to their servers, and charges per training run.
| Feature | Soup | HF AutoTrain |
|---|---|---|
| License | Apache-2.0 | Proprietary |
| Self-hosting | Yes | No (cloud-only) |
| Dataset privacy | Stays local | Uploaded to HF |
| Config format | soup.yaml | Web UI or YAML |
| Release gating | soup ship verdict | Manual |
| Migration support | From LLaMA-Factory, Axolotl, Unsloth | None |
Soup's soup ship release gate, semantic data dedup, and compliance templates (HIPAA, SOC2, EU-AI-Act) have no equivalent in AutoTrain. Teams that need auditable, reproducible training pipelines with signed provenance will find Soup more capable. AutoTrain is more accessible for one-off fine-tuning without any local infrastructure.
Soup vs Google Cloud Vertex AI
Vertex AI is a full ML platform covering model training, serving, and MLOps at enterprise scale. Compared to Vertex AI, Soup is narrower in scope (fine-tuning and post-training, not the full ML lifecycle) but requires no GCP account, no cloud spend, and no data egress. Vertex AI is the better choice for organizations already committed to GCP that need managed training at scale, AutoML pipelines, or integrated model monitoring. Soup is the better fit for teams that want self-hosted training on their own hardware with no vendor dependency.
Install and self-host#
Install Soup via pip with the training extra, then run soup init to create a config and soup train to begin training.
```bash
pip install "soup-cli[train]"
soup init
soup train
```What it's built on#
- Languages
- Python
- Frameworks
- FastAPIPyTorch
FAQ#
Can Soup really fine-tune an 8B model on a 4 GB GPU?
Yes, with layer streaming enabled (stream_layers: true in soup.yaml), which is currently BETA. The approach streams the frozen base model from CPU RAM or NVMe one decoder layer at a time instead of loading it fully into VRAM. Measured result: Llama-3.1-8B-Instruct at 119.6 tok/s in 3.32 GB peak on an RTX 3050 Laptop 4 GB. The measurement was taken on v0.72.2; a v0.73.0 correctness repair cost 4.8% throughput at 32B and has not been re-run on a 4 GB card. You can verify it yourself on a free Colab T4 using the published notebook in the repo.
Is Soup free to use commercially?
Yes. Soup is Apache-2.0 licensed, which permits commercial use, modification, and distribution without source-publishing obligations. There is no paid tier required for production use, no per-run API fee, and no telemetry. Training runs stay entirely on your own hardware.
How does Soup compare to Together AI or Hugging Face AutoTrain?
Together AI and Hugging Face AutoTrain are managed cloud services: you upload your data, they run training on their infrastructure, and you pay per GPU-hour. Soup is a self-hosted CLI: you run it on your own GPU, your data never leaves your machine, and the per-run cost is electricity. The tradeoff is that you manage the server and GPU, and some features (multi-GPU at scale, Apple Silicon) require additional setup. For teams with data-privacy requirements or tight cloud budgets, Soup is the stronger fit.
What Python versions does Soup support?
Python 3.10, 3.11, and 3.12. Python 3.13 is not supported because the PyTorch stack has not been validated on that version; pip resolves untested wheels that crash in the native extension before Soup starts. Use a virtual environment pinned to a supported version if your system Python is newer.
Can I migrate from LLaMA-Factory or Axolotl to Soup?
Yes. soup migrate --from llamafactory config.yaml or soup migrate --from axolotl config.yml converts your existing config to soup.yaml format in one command. Soup auto-detects parameters it can infer and fills them in. Review the output config before your first training run to catch anything the converter could not map, especially if your original config used non-standard PEFT variants or custom data pipelines.
Similar open-source tools#
ODS
Turn any computer into a private AI server
open-notebook
Self-host private AI research notebooks
Ollama
Run large language models locally on Mac, Linux, or Windows
Unsloth
Train LLMs locally without code using a browser-based interface
mTarsier
Free desktop app for managing MCP servers and AI agents
N8N2MCP
Bridge n8n automations into MCP tools for Claude and Cursor