LLM Foundry

LLM Foundry is MosaicML's open source framework for training, fine-tuning, and instruction-tuning large language models — the same codebase used to train MPT-7B and DBRX — designed for efficient GPU utilization and YAML-driven configuration that removes boilerplate from distributed training runs.

The Problem

Fine-tuning a large language model on custom data is harder than it should be. Hugging Face's transformers trainer works for small single-GPU experiments but is not optimized for multi-GPU efficiency at scale. Training a 70B model on proprietary data requires model parallelism, gradient checkpointing, and careful batch configuration that default trainers handle poorly. Closed platforms like OpenAI's fine-tuning API are simpler but send your training data to external servers and lock you into their model versions.

How LLM Foundry Solves It

LLM Foundry uses YAML-defined training jobs that handle distributed training, mixed-precision, and FlashAttention-2 automatically. Streaming datasets load directly from S3 or GCS, so you do not need to download your entire training corpus to local disk. Full fine-tunes, LoRA, and instruction tuning from chat-formatted datasets are first-class workflows with example configs in the repository.

Key Features

• YAML-driven training configuration: change model, dataset, and hardware without Python edits
• FlashAttention-2 for memory-efficient attention at long context lengths
• Streaming datasets: load training data directly from S3 or GCS without local disk
• Full fine-tune, LoRA, and instruction tuning (chat format) as first-class workflows
• Compatible with Llama 3, Mistral, Falcon, DBRX, and MPT model families
• Composer integration for callbacks, early stopping, and experiment tracking

Who It's For

LLM Foundry is best for ML teams fine-tuning large language models on proprietary datasets without sending data to an external provider, organizations building domain-specific LLMs for legal, medical, or financial applications, and researchers reproducing or extending MosaicML's published model training runs.

Compared to Hugging Face Trainer

Unlike the Hugging Face transformers trainer, LLM Foundry is optimized for large-scale fine-tuning — FlashAttention-2, streaming datasets from object storage, and FSDP/Megatron parallelism are built-in defaults rather than manual configuration steps.