A reinforcement learning framework for training bipedal humanoid robots to walk, combining MuJoCo simulation with ROS 2 Humble deployment pipelines. Trained policies transfer from simulation to real hardware via domain randomization and system identification.

This project implements end-to-end reinforcement learning for humanoid locomotion:

• Training — PPO-based policy learning with parallel environments and reward shaping for stable bipedal walking
• Sim-to-Sim Validation — Transfer trained policies to MuJoCo for physics cross-validation before hardware deployment
• Sim-to-Real Transfer — Domain randomization, actuator modeling, and system identification for zero-shot transfer
• ROS 2 Deployment — Integration with Gazebo Sim and real robot control stacks via ROS 2 Humble

rl-bipedal-walking/
├── humanoid_descriptions/           # Vendored humanoid robot sources
│   ├── ros/                         # ROS/Gazebo-oriented packages
│   ├── rl/                          # RL training stacks from official sources
│   └── urdf_only/                   # Description-focused robot packages
├── humanoid/                         # RL training package
│   ├── envs/
│   │   ├── base/                     # Base legged robot environment
│   │   └── custom/
│   │       ├── humanoid_env.py       # Humanoid env (rewards, obs, domain rand)
│   │       └── humanoid_config.py    # Training & environment hyperparameters
│   ├── scripts/
│   │   ├── train.py                  # Launch RL training
│   │   ├── play.py                   # Visualize & export trained policy
│   │   └── sim2sim.py                # MuJoCo sim-to-sim validation
│   ├── algo/                         # PPO implementation
│   └── utils/                        # Logging, terrain, task registry
├── resources/
│   └── robots/
│       └── XBot/                     # Humanoid URDF, MJCF, meshes
│           ├── urdf/                 # URDF robot description
│           └── mjcf/                 # MuJoCo XML models
├── ros2_ws/                          # ROS 2 Humble workspace
│   └── src/
│       └── bipedal_robot_description/
│           ├── urdf/                 # Robot description
│           ├── launch/               # Gazebo Sim launch files
│           └── config/               # RViz configs
├── scripts/                          # Shell helper scripts
├── logs/                             # Training run logs & exported policies
├── setup.py                          # Package install
└── requirements.txt

• The main RL stack in humanoid/ is the primary training code in this repo today.
• The ROS 2 workspace in ros2_ws/ can spawn the current humanoid model in Gazebo Sim, but it is not a full ros2_control stack yet.
• Official external humanoid sources are mirrored under humanoid_descriptions/ so you can swap in stronger robot descriptions without adding nested Git repos.

• Ubuntu 22.04
• Python 3.8+
• NVIDIA GPU with CUDA 11.x+
• ROS 2 Humble
• MuJoCo 2.3.6+

git clone https://github.com/darshmenon/rl-bipedal-walking
cd rl-bipedal-walking

# Create virtual environment
conda create -n humanoid-rl python=3.8
conda activate humanoid-rl

# Install PyTorch with CUDA
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia

# Install the package
pip install -e .
pip install -r requirements.txt

# PPO training with 4096 parallel environments
python humanoid/scripts/train.py --task humanoid_ppo --run_name v1 --headless --num_envs 4096

Training runs with parallel environments. Policy checkpoints are saved to logs/.

train.py requires Isaac Gym, which is discontinued and may not be installable on newer setups. humanoid/scripts/train_mujoco.py is an Isaac-Gym-free alternative that trains the same XBot-L model directly in MuJoCo with stable-baselines3 PPO:

python humanoid/scripts/train_mujoco.py --run_name v1 --num_envs 16 --total_timesteps 2000000

Exported policies from either path are loadable by sim2sim.py.

# Load trained policy and export JIT model
python humanoid/scripts/play.py --task humanoid_ppo --run_name v1

This exports a JIT-compiled policy to logs/<experiment>/exported/policies/ for deployment.

# Run trained policy in MuJoCo for validation
python humanoid/scripts/sim2sim.py --load_model logs/XBot_ppo/exported/policies/policy_1.pt

# With terrain
python humanoid/scripts/sim2sim.py --load_model logs/XBot_ppo/exported/policies/policy_1.pt --terrain

Validates the trained policy under MuJoCo physics before deploying to hardware.

# Build ROS 2 workspace
cd ros2_ws
source /opt/ros/humble/setup.bash
colcon build --symlink-install
source install/setup.bash

# Launch Gazebo Sim with humanoid robot
ros2 launch bipedal_robot_description spawn_robot.launch.py

To launch the alternate Gazebo entrypoint:

ros2 launch bipedal_robot_description gazebo_launch.py

The primary training algorithm uses an actor-critic architecture with the following specs:

The reward is a weighted sum of locomotion objectives:

• Velocity tracking — Forward/lateral/angular velocity command following
• Gait phase — Foot contact timing aligned to reference sinusoidal gait
• Joint position tracking — Penalize deviation from reference motion
• Stability — Base orientation, height, and acceleration penalties
• Feet clearance — Swing leg lift during gait cycle
• Energy efficiency — Torque, joint velocity, and action smoothness penalties
• Collision — Penalize undesired body contacts

Applied during training for robust sim-to-real transfer:

• Friction: [0.1, 2.0]
• Base mass: [-5.0, +5.0] kg
• Random pushes every 4s (linear + angular)
• Action delay and noise injection

Default: XBot-L — a 1.65m humanoid with 12 DOF legs.

Robot assets live in resources/robots/XBot/ with both URDF and MJCF descriptions.

The sim2sim.py script enables zero-shot policy transfer between simulators:

1. Train in parallel simulation → export JIT policy
2. Load policy in MuJoCo with matched robot model
3. Run PD control loop at 1000 Hz (policy at 100 Hz via decimation)
4. Verify walking behavior, contact forces, and stability

This catches policy brittleness and physics mismatches before real hardware deployment.

The ros2_ws provides deployment infrastructure on ROS 2 Humble:

• URDF/SDF humanoid robot description
• Gazebo Sim launch files with ROS-GZ bridge
• Joint state publisher and robot_state_publisher
• RViz visualization config
• Scaffolding for policy inference node (trained model → joint torque commands)
• A lightweight spawn path for testing descriptions before integrating controllers

Current limitation:

ros2_control is not fully wired into the local bipedal_robot_description package yet, so the ROS workspace is currently better suited for description validation and spawn testing than for full controller bring-up.

# Key ROS 2 topics
/joint_states          # sensor_msgs/JointState
/odom                  # nav_msgs/Odometry
/cmd_vel               # geometry_msgs/Twist

There are two different simulation paths in this repo:

• ros2_ws/ uses ROS 2 Humble + Gazebo Sim
• humanoid_descriptions/ros/unitree_ros and humanoid_descriptions/urdf_only/berkeley_humanoid_description are imported from ROS 1 + classic Gazebo ecosystems

Use this when you want to validate the repo's current ROS 2 spawn flow:

cd ros2_ws
source /opt/ros/humble/setup.bash
colcon build --symlink-install
source install/setup.bash

ros2 launch bipedal_robot_description spawn_robot.launch.py

Alternate Gazebo entrypoint:

ros2 launch bipedal_robot_description gazebo_launch.py

The imported Unitree packages live in:

• humanoid_descriptions/ros/unitree_ros/robots/g1_description
• humanoid_descriptions/ros/unitree_ros/robots/h1_description
• humanoid_descriptions/ros/unitree_ros/robots/h1_2_description
• humanoid_descriptions/ros/unitree_ros/robots/h2_description
• humanoid_descriptions/ros/unitree_ros/robots/r1_description
• humanoid_descriptions/ros/unitree_ros/robots/r1_air_description

These are ROS 1 packages. Start with the one that already includes a direct Gazebo launch:

cd humanoid_descriptions/ros/unitree_ros
# inside a ROS 1 catkin workspace
roslaunch h1_description gazebo.launch

For quick visual checks without Gazebo:

roslaunch h1_description display.launch

The imported unitree_gazebo and unitree_controller packages are also included for classic Gazebo and low-level controller experiments.

Important note:

The upstream Unitree stack is aimed at ROS 1 and classic Gazebo, not the local ROS 2 Gazebo Sim workspace in this repo.

For a ROS 2 display, a wrapper package is included at ros2_ws/src/h1_description:

cd ros2_ws
source /opt/ros/humble/setup.bash
colcon build --packages-select h1_description --symlink-install
source install/setup.bash

ros2 launch h1_description display.launch.py

The Berkeley package is imported under:

• humanoid_descriptions/urdf_only/berkeley_humanoid_description

Classic Gazebo test:

# inside a ROS 1 catkin workspace
roslaunch berkeley_humanoid_description empty_world.launch

Standalone URDF/RViz test:

roslaunch berkeley_humanoid_description standalone.launch

These imports are best treated as description sources first:

• humanoid_descriptions/urdf_only/booster_assets
• humanoid_descriptions/urdf_only/robotera_models

Useful files include:

• humanoid_descriptions/urdf_only/booster_assets/robots/T1/T1_23dof.urdf
• humanoid_descriptions/urdf_only/booster_assets/robots/K1/K1_22dof.urdf
• humanoid_descriptions/urdf_only/robotera_models/star1

They do not yet come with a ready-to-run local ROS 2 spawn wrapper in this repo, so the usual next step is:

1. Copy the chosen URDF and meshes into a ROS package
2. Point robot_state_publisher at that URDF
3. Spawn it with either ros_gz_sim create in ROS 2 or gazebo_ros spawn_model in ROS 1

If you want, the next integration step is to replace ros2_ws/src/bipedal_robot_description/urdf/bipedal.urdf with one of these imported robots and add a dedicated launch package for it.

1. Add URDF and MJCF assets to resources/robots/<your_robot>/
2. Create a config in humanoid/envs/custom/ inheriting from LeggedRobotCfg
3. Set asset path, body names, default joint angles, and PD gains
4. Register the task in humanoid/envs/__init__.py
5. Update sim2sim.py joint mapping if needed

If you want to start from an existing robot instead of creating one from scratch, check humanoid_descriptions/ first. The vendored sources include official Unitree stacks plus Berkeley, Booster, and RobotEra description packages.

• Headless mode: Use --headless for faster training without rendering
• GPU selection: --sim_device=cuda:0 --rl_device=0
• Resume training: Set resume=True and load_run in config
• Terrain curriculum: Enable mesh_type='trimesh' in config for rough terrain training
• Monitoring: Training logs are compatible with TensorBoard and Weights & Biases

Located in humanoid/envs/custom/humanoid_config.py:

# Training
num_envs = 4096
max_iterations = 3000
episode_length_s = 24

# Domain Randomization
randomize_friction = True       # [0.1, 2.0]
randomize_base_mass = True      # [-5, +5] kg
push_robots = True              # random impulses
action_delay = 0.5              # simulate actuator lag
action_noise = 0.02             # observation noise

# Control
action_scale = 0.25
decimation = 10                 # 1000Hz sim → 100Hz policy

• Humanoid-Gym: Zero-Shot Sim2Real Transfer — RobotEra / Tsinghua
• Advancing Humanoid Locomotion with Denoising World Model Learning — RSS 2024
• legged_gym — ETH Zurich RSL
• MuJoCo Menagerie — DeepMind

BSD-3-Clause License