Who miles is for#
AI labs training frontier LLMs with RLHF or GRPO
Miles supports GRPO, GSPO, PPO, and REINFORCE++ out of the box, with customizable reward functions, advantage computation, and policy-lag controls. Agentic multi-turn rollouts use the TITO data path to preserve token fidelity. Periodic Labs used Miles to train trillion-parameter models across thousands of GPUs.
Skip if:
Skip if you are fine-tuning a small model (under 7B parameters) on a single GPU. Miles is designed for distributed multi-GPU workloads where async scheduling and P2P weight synchronization provide measurable benefit.
Enterprises building coding or computer-use agents
Miles integrates with agent harnesses including Harbor, HUD, NeMo Gym, OpenEnv, and Verifiers, and supports task sandboxes on Daytona, E2B, Modal, and AgentENV. IBM and Decagon have cited Miles for agentic RL across coding, computer use, and non-verifiable reward domains at scale.
Skip if:
Skip if your agent training involves prompting a hosted model API rather than weight updates. Miles trains model parameters; it is not a prompt orchestration or agent routing framework.
Research teams experimenting with on-policy distillation
On-policy distillation (OPD) is a supported recipe in Miles alongside SFT and RL. The same rollout infrastructure that handles agentic RL also drives distillation runs, so teams can switch between training objectives without rebuilding their pipeline infrastructure.
Skip if:
Skip if your distillation workflow uses only offline data. Miles is built for on-policy and hybrid schedules where rollout and training run concurrently.
Teams needing diffusion model post-training with RL
Miles-diffusion (a companion repository at github.com/radixark/miles_diffusion) supports Flow-GRPO, DiffusionNFT, and SFT on an SGLang-diffusion rollout engine with an FSDP2 trainer. Supported diffusion models include Stable Diffusion 3/3.5, Wan 2.2, and LTX-2.
Skip if:
Skip if your diffusion workflow does not involve reinforcement learning or supervised fine-tuning. Miles-diffusion is a post-training tool, not a diffusion inference framework.
The problem it solves#
Running reinforcement learning post-training for frontier-scale LLMs is a hard systems engineering challenge. Rollout generation and training must run concurrently without idle GPU time, but keeping model weights synchronized across large distributed clusters is expensive. Agentic workloads add further complexity: multi-turn trajectories with tool calls and environment interactions must preserve exact token metadata to avoid training on corrupted inputs. MoE models introduce routing mismatches between rollout and training that destabilize long runs.
Teams that need custom reward functions, multi-turn agentic rollouts, or novel RL algorithms face limited options. Managed cloud training platforms (AWS SageMaker, Google Vertex AI) abstract the training loop in ways that make customization difficult and charge per-instance-hour. Building infrastructure from scratch is expensive and slow. Most teams either accept the constraints of a managed service, pay for proprietary training APIs, or maintain fragile custom systems that break when model architectures change.
How it solves it#
Fully async RL with sample-level scheduling
Rollout and training workers run concurrently, with a sample-level scheduler that replaces completed trajectories immediately. A bounded data buffer lets generation and optimizer steps proceed in parallel even when trajectory lengths vary, reducing GPU idle time on long-horizon agentic workloads.
Token-in-Token-Out (TITO) agentic rollout
Preserves the exact tokens and metadata produced during rollout across multi-turn agent trajectories containing tool calls, observations, and environment interactions. No detokenize/retokenize round-trip between rollout and training, which eliminates a common source of training data corruption in agentic RL.
P2P RDMA weight synchronization
Pushes new model weights from training workers to rollout engines in seconds, even on trillion-parameter models. P2P RDMA is the fast path for disaggregated setups. Periodic Labs running on Miles improved weight synchronization by 10x and weight conversion by 30x using this path.
Multi-backend distributed training
Supports Megatron-LM for large-scale tensor and pipeline parallelism, and PyTorch FSDP2 for teams that want to train HuggingFace model implementations directly. Low-precision training with MXFP8, NVFP4, FP8, INT4 QAT, BF16, and FP16 is available to reduce memory footprint on large runs.
Day-0 frontier model support
Verified support for frontier model families including DeepSeek, Kimi, GLM, Qwen, Inkling, and Nemotron from their release day. Covers dense, MoE, and multimodal architectures. Hardware support spans NVIDIA Blackwell, Hopper, and Ampere, plus AMD Instinct (MI300X through MI355X) via ROCm.
Agentic environment integrations
Connects to agent harnesses and environments including Harbor, HUD, NeMo Gym, OpenEnv, and Verifiers. Task sandboxes on AgentENV, Daytona, E2B, and Modal are supported. Custom rollout, reward, and data-processing functions can be added without modifying the core training loop.
Strengths and trade-offs#
Strengths
- Fault-tolerant rollout clusterWhen an SGLang rollout engine fails during a run, Miles recovers it and resumes in place without restarting or pausing the overall training run. The rollout router health-checks the engine fleet and reroutes requests automatically, an important advantage on long-horizon runs where a single engine failure would otherwise waste hours of training.
- Apache-2.0 license with no usage restrictionsApache-2.0 means you can run Miles on internal infrastructure, modify it for proprietary workloads, and distribute it commercially without restrictions. Your datasets, trajectories, checkpoints, and model weights stay on your own hardware. No per-GPU licensing fees, no data egress to a vendor API.
- Rollout Routing Replay (R3) for MoE training stabilityRecords expert routing decisions made during rollout and replays them in the trainer's forward pass, eliminating the MoE routing mismatch that causes instability on large runs. Compute and communication are overlapped to keep overhead low. Especially relevant for MoE models like DeepSeek-V4 and Qwen3 variants.
- Day-0 model coverage maintained at frontier paceMiles added day-0 support for DeepSeek-V4, Kimi-K3, Inkling, and Nemotron on their respective release days. Coverage is maintained through co-evolution with SGLang and partnerships with hardware vendors and model labs, so the training stack rarely lags behind newly released frontier models.
Trade-offs
- -GPU infrastructure required: no managed cloud tierMiles runs entirely on your own hardware. NVIDIA or AMD accelerators are required. There is no managed cloud option or hosted API tier. Teams without existing GPU clusters will need to provision infrastructure before running a training job, which adds upfront setup work compared to managed ML training services.
- -955 open GitHub issues as of September 2026With 955 open issues, the project is actively evolving and some rough edges remain. Teams running workloads that deviate significantly from the documented recipes (GRPO on supported models) may encounter undocumented behavior or configuration edge cases that require debugging against the GitHub issue tracker.
- -Full feature set requires Megatron-LM backendThe full parallelism, precision, and recipe support lives on the Megatron-LM backend. Teams that prefer the HuggingFace model implementation path (FSDP2) will have a simpler setup but will miss some of the most advanced features, as noted explicitly in the Miles documentation.
miles vs alternatives#
Miles vs AWS SageMaker
Both Miles and AWS SageMaker can run distributed LLM training at scale, but they target different deployment models. SageMaker is a managed cloud service with per-instance-hour pricing and no self-hosting option. Miles runs on your own GPU infrastructure under Apache-2.0 with no usage fees beyond compute.
| Feature | Miles | AWS SageMaker |
|---|---|---|
| License | Apache-2.0 | Proprietary |
| Self-hosting | Yes (required) | No (cloud-only) |
| RL algorithm support | GRPO, GSPO, PPO, REINFORCE++ | Custom code required |
| Agentic RL environments | Native integrations (Harbor, E2B, Modal) | Custom implementation required |
| Pricing | Infrastructure cost only | Per-instance-hour plus data transfer |
Miles is the better choice when you need specific RL algorithms, agentic rollout with multi-turn trajectories, or the ability to modify the training loop for novel research. SageMaker is the better choice when your team needs a managed environment with AWS IAM, VPC, and SageMaker Experiments for tracking, and prefers not to operate GPU cluster infrastructure.
Miles vs Google Vertex AI
Vertex AI offers managed training jobs with TPU and GPU support. Like SageMaker, it abstracts the distributed training infrastructure but provides less control over the training loop. Miles supports MXFP8, NVFP4, and INT4 QAT precision modes and includes Rollout Routing Replay (R3) for MoE training stability, neither of which is available on Vertex AI natively.
| Feature | Miles | Google Vertex AI |
|---|---|---|
| License | Apache-2.0 | Proprietary |
| MoE training stability | R3 routing replay | Not available natively |
| Low-precision training | MXFP8, NVFP4, FP8, INT4 QAT | BF16/FP16 standard |
| Agentic RL environments | 6+ native integrations | Custom only |
| Hardware | NVIDIA plus AMD ROCm | NVIDIA plus TPU |
Miles fits teams that train frontier models with specific RL algorithms, need reproducible agentic rollouts, or run on on-premises NVIDIA or AMD hardware. Vertex AI fits teams that want Google Cloud integration, AutoML, and model registry features in a fully managed pipeline without operating their own cluster.
What it's built on#
- Languages
- JavaScriptPython
FAQ#
Does Miles support running on AMD GPUs?
Yes. Miles supports AMD Instinct accelerators (MI300X, MI325, MI350, MI355X) via ROCm. DeepSeek-V4 Flash RL training on AMD Instinct MI355X is documented in the Miles blog. Hardware support and container images for each GPU are listed in the installation documentation.
Which RL algorithms does Miles support?
Miles supports GRPO, GSPO, PPO, and REINFORCE++ for reinforcement learning, plus SFT and on-policy distillation. Reward functions, advantage computation, loss, data selection, and policy-lag controls are all customizable, making it possible to implement new research algorithms without modifying the core training loop.
Can Miles work with an existing agent harness or environment?
Yes. Miles provides integrations for Harbor, HUD, NeMo Gym, OpenEnv, and Verifiers, along with sandbox providers including Daytona, E2B, Modal, and AgentENV. Custom rollout, reward, and data-processing functions can also be added without modifying the core training loop.
How does Miles handle failures during a long training run?
When an SGLang rollout engine fails, Miles recovers it and resumes the run in place without a restart or pause. The rollout router health-checks the engine fleet and reroutes requests during recovery. Fault tolerance documentation is in the advanced section of the Miles docs.
Does Miles support diffusion model training in addition to LLMs?
LLM and VLM post-training are the primary focus. Diffusion model support (Flow-GRPO, DiffusionNFT, and SFT) is available in Miles-diffusion, a companion project at github.com/radixark/miles_diffusion, which uses an SGLang-diffusion rollout engine and an FSDP2 trainer. Supported models include Stable Diffusion 3/3.5, Wan 2.2, and LTX-2.
Similar open-source tools#
fx
Native Zig CLI coding agent, 7.8 MiB, any AI model.
graphify
Turn your codebase into a knowledge graph your AI can trace and cite.
RAGMeUp
Open source RAG framework for any dataset or LLM
RealtimeSTT
Real-time speech-to-text library with VAD and wake words
code-graph-rag
AI-powered codebase analysis with knowledge graphs
Embabel
Agentic AI framework for the JVM