README

CleanRL (Clean Implementation of RL Algorithms)













CleanRL is a Deep Reinforcement Learning library that provides high-quality single-file implementation with research-friendly features. The implementation is clean and simple, yet we can scale it to run thousands of experiments using AWS Batch. The highlight features of CleanRL are:



 📜 Single-file implementation
    Every detail about an algorithm variant is put into a single standalone file. 
    For example, our ppo_atari.py only has 340 lines of code but contains all implementation details on how PPO works with Atari games, so it is a great reference implementation to read for folks who do not wish to read an entire modular library.
 📊 Benchmarked Implementation (7+ algorithms and 34+ games at https://benchmark.cleanrl.dev)
 📈 Tensorboard Logging
 🪛 Local Reproducibility via Seeding
 🎮 Videos of Gameplay Capturing
 🧫 Experiment Management with Weights and Biases
 💸 Cloud Integration with docker and AWS 

You can read more about CleanRL in our JMLR paper and documentation.

Notable CleanRL-related projects:

 corl-team/CORL: Offline RL algorithm implemented in CleanRL style
 pytorch-labs/LeanRL: Fast optimized PyTorch implementation of CleanRL RL algorithms using CUDAGraphs.


 ℹ️ Support for Gymnasium: Farama-Foundation/Gymnasium is the next generation of openai/gym that will continue to be maintained and introduce new features. Please see their announcement for further detail. We are migrating to gymnasium and the progress can be tracked in vwxyzjn/cleanrl277. 


 ⚠️ NOTE: CleanRL is not a modular library and therefore it is not meant to be imported. At the cost of duplicate code, we make all implementation details of a DRL algorithm variant easy to understand, so CleanRL comes with its own pros and cons. You should consider using CleanRL if you want to 1) understand all implementation details of an algorithm's variant or 2) prototype advanced features that other modular DRL libraries do not support (CleanRL has minimal lines of code so it gives you great debugging experience and you don't have do a lot of subclassing like sometimes in modular DRL libraries).

 Get started

Prerequisites:
 Python =3.7.1,<3.11
 uv 0.7.9+

To run experiments locally, give the following a try:

bash
git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl
uv pip install .

 alternatively, you could use uv venv and do
 python run cleanrl/ppo.py
uv run python cleanrl/ppo.py \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000

 open another terminal and enter cd cleanrl/cleanrl
tensorboard --logdir runs


To use experiment tracking with wandb, run
bash
wandb login  only required for the first time
uv run python cleanrl/ppo.py \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000 \
    --track \
    --wandb-project-name cleanrltest


If you are not using uv, you can install CleanRL with requirements.txt:

bash
 core dependencies
pip install -r requirements/req