Who Model-Optimizer is for#
ML engineers reducing inference cost on deployed LLMs
ModelOpt's PTQ path quantizes a deployed Hugging Face or Megatron model to FP8 or NVFP4 in a single calibration pass, cutting memory footprint 2x to 4x and enabling higher throughput on the same hardware. The export to vLLM or TensorRT-LLM is a direct pipeline step with no manual format work.
Skip if:
Skip if your inference runtime is not TensorRT-LLM, vLLM, or SGLang. ModelOpt's export paths are tied to NVIDIA deployment frameworks; optimization results do not directly transfer to other runtimes.
Research teams building compressed model variants
Minitron-style pruning combined with two-phase distillation lets research teams produce smaller model variants from an existing checkpoint. Puzzletron adds heterogeneous NAS, and the result can be published as a smaller, faster model with documented quality retention, as Bielik.AI did with Bielik Minitron 7B.
Skip if:
Skip if your base model is not in ModelOpt's support matrix. Check the pruning and distillation support matrices in the GitHub repo before starting; architectures outside the matrix require custom integration work.
Organizations requiring on-premises model optimization
Because ModelOpt runs inside your own infrastructure with no external API calls during the optimization step, it fits environments where model weights and training data cannot leave the organization's network. The Apache-2.0 license permits internal commercial use without licensing agreements or usage-based billing.
Skip if:
Skip if you have no NVIDIA GPU infrastructure. ModelOpt's hardware-specific quantization and export paths require CUDA-capable hardware for the optimization step.
Inference teams deploying diffusion models
ModelOpt supports quantization for Hugging Face diffusers models, and NVIDIA has documented a 60% diffusion latency reduction using ModelOpt with TensorRT in an Adobe deployment. Cache Diffusion is an additional ModelOpt technique for further inference acceleration on diffusion architectures.
Skip if:
Skip if your deployment target is not NVIDIA TensorRT. Diffusion model optimizations in ModelOpt are coupled to the TensorRT export path.
The problem it solves#
Deploying large language models at production latency and acceptable cost is a consistent pain point for engineering teams. A 70B parameter model requires substantial GPU memory and compute; running inference on it at scale means either buying more hardware than the budget allows or accepting latency that makes the product unusable.
The standard path has been to rely on paid compression services or to accept that model optimization is too complex to build in-house. Post-training quantization, pruning, and distillation each require care to avoid accuracy degradation, and applying them in combination while targeting a specific deployment framework adds further complexity. Teams end up with underutilized hardware, oversized cloud bills, or models that never make it to production at all.
How it solves it#
Post-Training Quantization
Compresses model size 2x to 4x by reducing weight precision from FP16/BF16 to FP8, INT8, INT4, or NVFP4, with minimal accuracy loss on most architectures. Works on Hugging Face LLMs and VLMs, diffusion models, and ONNX graphs. PTQ requires no retraining and runs in a single pass over a calibration dataset.
Quantization-Aware Distillation
Recovers the accuracy that aggressive quantization costs by running a few training steps with the original model as teacher. QAD on Qwen3.6-35B-A3B with W4A4 NVFP4 achieved up to 1.30x vLLM throughput over BF16 at 3.1x smaller checkpoint size while matching BF16 accuracy. Supports Megatron-Bridge and Hugging Face training frameworks.
Pruning and Neural Architecture Search
Reduces model parameters or memory footprint by removing unnecessary weights. Includes Minitron-style pruning plus distillation, used to produce Bielik Minitron 7B (33% smaller, 50% faster, 90% quality retained) and to compress Colosseum-355B to 260B. Puzzletron adds heterogeneous pruning and NAS for LLMs and VLMs.
Speculative Decoding
Trains lightweight draft modules that predict extra tokens during the main model's inference pass, reducing wall-clock latency per token. ModelOpt supports vanilla Markov heads (DSpark) and GRU-based correction heads (Domino) on the shared DFlash backbone. Works with Hugging Face and Megatron-LM training frameworks.
Multi-Framework Deployment Export
Exports quantized checkpoints in formats ready for TensorRT-LLM, TensorRT, vLLM, and SGLang without manual format conversion. The unified Hugging Face export API covers both transformers and diffusers models. Pre-quantized checkpoints for DeepSeek-R1, Llama 3.x, and Nemotron 3 are available on Hugging Face.
Strengths and trade-offs#
Strengths
- Apache-2.0 license with no usage restrictionsThe full library is Apache-2.0 licensed, permitting unrestricted commercial and research use. You can run ModelOpt on your infrastructure, modify it, and distribute derivative work without royalties or commercial-use clauses. This contrasts with paid optimization services where the optimization pipeline is proprietary and usage is metered.
- Native NVIDIA ecosystem integrationModelOpt is developed by NVIDIA and integrates directly with TensorRT-LLM, TensorRT, Megatron-Bridge, and vLLM deployment paths. Optimized checkpoints move into production without format conversion steps. Adobe used ModelOpt with TensorRT to achieve a 60% reduction in diffusion latency and a 40% reduction in total cost of ownership.
- Pre-quantized checkpoint collection on Hugging FaceNVIDIA publishes ready-to-deploy ModelOpt-optimized checkpoints for major models including DeepSeek-R1-FP4, Llama-3.3-70B-Instruct-FP4, Llama-3.1-405B-Instruct-FP4, and Nemotron 3 series. Teams that need inference acceleration without running the optimization pipeline themselves can use these directly on TensorRT-LLM or vLLM.
- Actively maintained by NVIDIA with a public roadmapModelOpt was open sourced in January 2025 and has been updated continuously, with the last push on September 26, 2026. NVIDIA ships new techniques as blog-backed algorithm releases. The roadmap is tracked publicly on GitHub, and 4,473 stars with 649 forks indicate active community use.
Trade-offs
- -Requires NVIDIA GPU hardwareMost optimization techniques and all deployment export paths are built for NVIDIA hardware. FP8, NVFP4, and INT4 quantization targets assume TensorRT or CUDA-capable GPUs. Teams running on AMD, Apple Silicon, or CPU-only inference will find ModelOpt's advanced techniques inapplicable, and the export path to TensorRT-LLM does not transfer to non-CUDA runtimes.
- -Pre-1.0 API with short deprecation windowModelOpt follows a 0.x versioning scheme with a one-release migration window after deprecations. Breaking changes to the Python API can appear in minor version updates while the library remains pre-1.0. Teams building automated optimization pipelines should pin the version and test upgrades before deploying changes.
- -418 open issues across supported model architecturesThe public issue tracker shows 418 open issues, reflecting active user pain points and gaps between documented support and real-world model architectures. Not all are bugs, but the volume indicates friction with specific models or quantization paths. Check the support matrix and open issues before committing to a specific model or technique.
Model-Optimizer vs alternatives#
Model Optimizer vs CLIKA
Model Optimizer and CLIKA both target the model compression and optimization problem. Model Optimizer is an Apache-2.0 open source Python library from NVIDIA, self-hosted and free to run. CLIKA is a commercial model optimization service.
| Feature | Model Optimizer | CLIKA |
|---|---|---|
| License | Apache-2.0 | Proprietary |
| Self-hosting | Yes | No (managed service) |
| Pricing | Free (self-hosted) | Paid subscription |
Model Optimizer is the stronger choice for teams deploying within the NVIDIA ecosystem. If your inference target is TensorRT-LLM, TensorRT, vLLM, or SGLang on NVIDIA GPUs, ModelOpt's export pipeline is direct and validated at scale: NVIDIA used it to optimize Nemotron 3 Ultra 550B, and Adobe used it with TensorRT to achieve a 60% reduction in diffusion latency and a 40% reduction in total cost of ownership. The Apache-2.0 license means no usage metering, no data sent to a third-party service, and no subscription cost.
CLIKA is worth considering when your team needs a fully managed optimization service with commercial support rather than an open source pipeline to operate in-house. Teams without dedicated NVIDIA GPU infrastructure or without ML expertise to run the optimization pipeline may find a managed service faster to onboard, trading operational control for convenience.
Quick start#
Install the stable release from PyPI with pip.
```bash
pip install -U nvidia-modelopt[all]
```What it's built on#
- Languages
- Python
FAQ#
Is NVIDIA Model Optimizer free to use?
Yes. ModelOpt is Apache-2.0 licensed and available on PyPI at no cost. You can use it commercially, modify it, and distribute derivative work without restrictions. The only requirements are NVIDIA GPU infrastructure with compatible CUDA and the target deployment framework installed.
What is the difference between PTQ and QAT in ModelOpt?
Post-training quantization (PTQ) reduces model weight precision in a single calibration pass with no retraining, giving 2x to 4x compression at some accuracy cost. Quantization-aware training (QAT) or quantization-aware distillation (QAD) adds a short training phase with the original model as teacher to recover accuracy. PTQ is faster to run; QAT and QAD produce higher-accuracy quantized models at the cost of additional compute.
Which inference frameworks does Model Optimizer support for export?
ModelOpt exports quantized checkpoints for TensorRT-LLM, TensorRT, vLLM, and SGLang. The unified Hugging Face export API covers both transformers and diffusers models. Pre-quantized checkpoints for DeepSeek-R1, Llama 3.x, and Nemotron 3 series are available on the NVIDIA Hugging Face collection.
Does Model Optimizer work with non-NVIDIA hardware?
ModelOpt is designed for NVIDIA hardware and its export paths target CUDA-based deployment frameworks. FP8, NVFP4, and INT4 quantization techniques assume NVIDIA GPUs. Teams deploying on AMD, Apple Silicon, or CPU-only setups will need a different optimization stack; ModelOpt does not currently cover those targets.
Can I use Model Optimizer on my own models, or only on NVIDIA's checkpoints?
ModelOpt accepts any Hugging Face, PyTorch, or ONNX model as input, but the depth of support varies by architecture. Each technique has its own support matrix published in the GitHub repo. Check the quantization, pruning, distillation, or speculative decoding support matrix for your specific architecture before starting to avoid blocked paths.
Similar open-source tools#
Hermes Agent
Self-hosted AI agent with persistent memory, multi-channel chat, and model choice across OpenAI, OpenRouter, and custom endpoints.
Pi
The minimal, self-extensible coding agent for the terminal
leanctx
Drop-in LLM token compression. Code stays verbatim; prose compresses.
token-optimizer-mcp
Token savings and persistent knowledge graphs for AI coding agents
llama-swap
Hot-swap AI models on your local inference server
magnitude
Local models for your agent, tuned for your hardware