Who microduck_rl is for#
Robotics researchers studying sim2real transfer
microduck_rl provides a documented, reproducible sim2real pipeline for a real hardware target. The actuator modeling depth (voltage control law, gear backlash, domain randomization) makes it a concrete reference for studying what physical fidelity is necessary and sufficient for successful policy transfer on servo-driven bipeds.
Skip if:
If your research target is a different hardware platform or a larger robot with different actuator characteristics, the environment stack does not generalize without significant rework to the MJCF models and actuator config.
Microduck owners extending the behavior library
The task registry is extensible: each task family is a separate environment config file. Owners who want to teach the robot a new motion can follow the existing task structure, implement reward functions in `mdp.py`, and train with the same pipeline used for the built-in 13 tasks.
Skip if:
If you do not own or plan to own Microduck hardware, there is no general-purpose benefit to this repository over other RL locomotion environments.
ML engineers evaluating RL deployment on embedded hardware
The ONNX export path with a baked normalizer and a shared observation contract demonstrates a concrete approach to deploying learned policies on a resource-constrained robot runtime. Engineers evaluating whether PPO-trained policies can run reliably on embedded hardware will find the export and inference scripts directly applicable.
Skip if:
Skip if your target platform uses a different deployment format (TensorRT, TFLite) or requires a format-agnostic inference pipeline; ONNX is the only supported export path here.
The problem it solves#
Building bipedal robot locomotion policies requires a simulation environment that faithfully replicates the physics of real servo joints. Off-the-shelf simulators treat actuators as ideal, which means policies that train well in simulation often fail on hardware when they encounter real friction, gear backlash, and voltage sag under load.
For small, low-cost bipedal robots like Microduck, the sim2real gap is especially pronounced. At this scale, tiny servos driving an 800 g frame are the primary source of discrepancy between simulation and physical behavior. Most existing robot RL tooling is built around larger platforms and uses simplified actuator models. Researchers who want sim2real transfer on a platform like Microduck have had to build the actuator modeling and domain randomization stack themselves.
How it solves it#
13 locomotion task environments
Covers walking, fall recovery, sit-to-stand, ball kicking, forward rolling (roulade), and roller-skating gaits on flat and rough terrain. Each task has a flat variant and most have a rough-terrain twin; roller tasks add three additional skate-specific environments. The task registry is live and queryable with `uv run list-envs`.
High-fidelity sim2real actuator modeling
Uses the BAM M6 model for the Dynamixel XL330, covering voltage control law, back-EMF, and Coulomb/Stribeck friction. Per-environment domain randomization varies battery voltage, voltage sag under load, command delay, and friction magnitude. A backlash variant adds ±1° of gear play to all 14 servo joints.
ONNX export with baked observation normalizer
The export script bakes the observation normalizer into the ONNX graph, so deployed policies always see correctly scaled inputs without any runtime preprocessing step. The shared 61-dimensional observation contract lets any trained policy hot-swap onto the robot without firmware changes.
PPO training on MuJoCo Warp
Policies train at 50 Hz using PPO through the mjlab framework on MuJoCo Warp. A usable walking gait takes roughly one to two hours at 4,096 parallel environments on a CUDA GPU. Checkpoint resume is one flag away on any train command.
Hugging Face Jobs offload for GPU-free setups
Teams without a local CUDA GPU can route any train command to Hugging Face Jobs by adding the `--hf-jobs` flag. This removes the GPU hardware requirement from the training step and lets the repository run on CPU-only machines for everything except policy training itself.
Hot-swappable runtime policies
Walk, fall-recovery, roulade, and sit-stand policies share a 61-dimensional actor observation layout. Environments that do not use a command slot zero-pad it rather than dropping it, keeping the ONNX export and robot runtime unchanged across the full policy library.
Strengths and trade-offs#
Strengths
- Complete sim2real recipe, not just a task listMost RL environment repos provide task definitions and leave the physics gap to the user. microduck_rl ships the full recipe: BAM actuator modeling, backlash simulation, per-environment domain randomization, and the reward-design lessons from the project documented in AGENTS.md. The gap between training result and hardware deployment is reproducible and explained.
- Apache 2.0 license with no usage costApache 2.0 allows commercial use, modification, and redistribution without royalties. Unlike NVIDIA Isaac Sim (proprietary, seat-based licensing) or AWS RoboMaker (pay-per-use cloud compute), the training environment itself carries no licensing fee.
- Auditable reward designAGENTS.md documents the reward-design rules distilled from the project, readable by both human engineers and AI coding agents working in the repository. Each task family is a separate environment config file with `ENABLE_*` booleans for domain randomization toggles, so reward functions are visible and modifiable rather than hidden in framework internals.
Trade-offs
- -Requires a CUDA GPU for trainingTraining runs through MuJoCo Warp, which requires a CUDA-capable GPU. On ARM-based machines (DGX Spark, Jetson), the first dependency sync pulls around 2 GB of CUDA wheels and can time out; the README documents the `UV_HTTP_TIMEOUT=600` workaround. Hugging Face Jobs offloads this requirement but adds account setup overhead.
- -Purpose-built for Microduck hardware onlyThe robot models, actuator configs, and task definitions target the Microduck 14-servo bipedal platform specifically. Adapting these environments to a different robot requires replacing the MJCF models, actuator config, and all joint-layout references throughout the codebase. This is not a general-purpose bipedal locomotion framework.
- -3D model files carry a non-commercial licenseThe code is Apache 2.0, but the 3D model files (MJCF exports from Onshape) are Creative Commons BY-SA-NC, which prohibits commercial use. Teams planning to build commercial hardware from the Microduck physical design need to review the CC license separately from the code license.
microduck_rl vs alternatives#
microduck_rl vs NVIDIA Isaac Sim
NVIDIA Isaac Sim is a full-featured, GPU-accelerated robot simulator built on Omniverse with PhysX physics and broad multi-robot support. microduck_rl is a focused training suite for one 25 cm bipedal platform, built on MuJoCo Warp with an actuator model specific to the Dynamixel XL330 servo.
| Feature | microduck_rl | NVIDIA Isaac Sim |
|---|---|---|
| License | Apache-2.0 | Proprietary (free for research use) |
| Self-hosting | Yes, requires CUDA GPU | Requires Omniverse/CUDA |
| Physics engine | MuJoCo Warp | PhysX (Omniverse) |
| Robot coverage | Microduck only | Multi-robot, extensible |
| Actuator modeling | Voltage-level BAM model | Simplified joint model |
Isaac Sim is the better choice when you need to simulate multiple robot types, integrate with NVIDIA's robotics stack, or use photorealistic rendering for domain randomization. microduck_rl wins when Microduck is your hardware target and you want the exact sim2real recipe, including backlash modeling and BAM actuator physics, at zero licensing cost.
microduck_rl vs AWS RoboMaker
AWS RoboMaker is a managed cloud simulation service built on ROS and Gazebo, with infrastructure provisioning and fleet management. It charges per compute hour and requires an AWS account. microduck_rl is self-hosted, Apache 2.0 licensed, and uses MuJoCo Warp rather than Gazebo.
| Feature | microduck_rl | AWS RoboMaker |
|---|---|---|
| License | Apache-2.0 | Proprietary (pay-per-use) |
| Infrastructure | Local CUDA GPU or Hugging Face Jobs | AWS cloud (metered) |
| Physics | MuJoCo Warp | Gazebo |
| Target platform | Microduck bipedal | Any ROS-compatible robot |
| Actuator fidelity | High (voltage-level BAM model) | Standard Gazebo plugins |
RoboMaker is worth considering when you need fleet simulation at scale with ROS integration and prefer not to manage GPU infrastructure. microduck_rl is the better fit when Microduck is your target, you want zero training cost, and MuJoCo physics fidelity matters more than ROS ecosystem integration.
microduck_rl vs Unity Robotics Hub
Unity Robotics Hub provides tooling for robot simulation in the Unity game engine, used for synthetic data generation and sim2real research with ML-Agents. MuJoCo Warp, which microduck_rl uses, achieves faster physics throughput for RL training at scale than Unity's PhysX-based engine.
| Feature | microduck_rl | Unity Robotics Hub |
|---|---|---|
| License | Apache-2.0 | Proprietary (Unity license) |
| Physics engine | MuJoCo Warp | PhysX (Unity) |
| Parallel env support | 4,096+ envs on one GPU | Limited by Unity architecture |
| Actuator modeling | High-fidelity BAM model | Basic joint controllers |
Unity Robotics is the better choice when photorealistic rendering, synthetic data generation, or a visual scripting environment matters to your workflow. microduck_rl wins on training throughput, actuator physics fidelity for servo-driven platforms, and licensing cost.
Install and self-host#
Install via git clone; training requires a CUDA GPU and the uv package manager.
```bash
git clone https://github.com/pollen-robotics/microduck_rl
cd microduck_rl
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096
```What it's built on#
- Languages
- Python
FAQ#
Does microduck_rl require actual Microduck hardware to use?
No. Training and policy visualization run entirely in simulation using MuJoCo. You need a CUDA GPU and the mjlab framework for training, but not the physical robot. Hardware is only required if you want to deploy and test the exported ONNX policy on the real Microduck.
How does microduck_rl compare to NVIDIA Isaac Sim for bipedal RL?
Isaac Sim is a general-purpose robot simulator with broad hardware support, PhysX-based physics, and proprietary licensing. microduck_rl is narrower: it targets Microduck specifically, uses MuJoCo Warp, and is Apache 2.0 licensed at no cost. Isaac Sim is the better choice for teams simulating multiple robot types or integrating with NVIDIA's broader robotics stack. microduck_rl is the better choice when Microduck is your hardware target and you want the exact actuator model and reward design that produced the published sim2real results.
Can I run training without a local CUDA GPU?
Yes. Add --hf-jobs to any train command to submit the run to Hugging Face Jobs instead of running locally. See scripts/hf/README.md for configuration. CPU-only config-invariant and reward-function regression tests are also available via pytest and do not require a GPU.
What is the sim2real gap strategy used in this repository?
microduck_rl addresses the sim2real gap through three mechanisms: BAM actuator modeling (voltage control law, back-EMF, and friction for the Dynamixel XL330), per-environment domain randomization on battery voltage, command delay, and friction magnitude, and explicit backlash simulation adding ±1° of gear play to all 14 servo joints. The observation normalizer is baked into the exported ONNX graph so policies see consistent inputs on real hardware.
Is microduck_rl free for commercial use?
The code and training environments are Apache 2.0 licensed, which allows commercial use, modification, and redistribution without royalties. The 3D model files (MJCF exports) carry a separate Creative Commons BY-SA-NC license, which prohibits commercial use of those assets. If your use case involves the physical robot design rather than just the training code, review both licenses before proceeding.
Similar open-source tools#
OpenCode
OpenCode is an open-source AI coding agent that assists developers in
monocode
One desktop UI for all your AI coding agents.
reverse-skill
AI skill router for reverse engineering and penetration testing
patent-disclosure-skill
Chinese patent disclosure drafting with AI-powered point mining
crawl4ai
LLM-ready web crawling without API keys or rate limits
heretic
Decensor local LLMs automatically with directional ablation