This repository is a collection of Python implementations of various Reinforcement Learning (RL) algorithms. The primary goal is educational: to get a deep and intuitive understanding of how these algorithms work under the hood. π§ Due to the recent explosion in the AI domain especially Large Language Models, and many more applications it is important to understand core reinforcement learning algorithms.
This repository also includes a comprehensive cheat sheet summarizing key concepts and algorithms for quick reference.
This is not a performance-optimized library! I prioritize readability and clarity over speed and advanced features. Think of it as your interactive textbook for RL.
| Date | Update |
|---|---|
| 16 November 2026 | Version 1.0.0. Every notebook re-run from the top after fixing bugs that changed what they teach: terminal states and the Cliff Walking rules in Q-learning, GAE and critic targets in PPO, A2C and TRPO, gradient accumulation in A3C, the MCTS backup, HAC's lost low-level experience, and more. New notebook 19 on bandits for dynamic pricing, plus tests and CI. See the changelog. |
| 2 April 2025 | Added a comprehensive RL Cheat Sheet summarizing all implemented algorithms and core concepts. Repository now includes 18 algorithm notebooks. |
| 30 March 2025 | Added 18 new algorithms. |
- Focus on Fundamentals: Learn the core logic without the abstraction of complex RL libraries. We use basic libraries (NumPy, Matplotlib, PyTorch) to get our hands dirty. π οΈ
- Beginner-Friendly: Step-by-step explanations guide you through each algorithm, even if you're new to RL. πΆ
- Interactive Learning: Jupyter Notebooks provide a playground for experimentation. Tweak hyperparameters, modify the code, and see what happens! π§ͺ
- Clear and Concise Code: We strive for readable code that closely mirrors the mathematical descriptions of the algorithms. No unnecessary complexity! π
- Quick Reference: Includes a detailed Cheat Sheet for fast lookups of formulas, pseudocode, and concepts.
The repository currently includes implementations of the following RL algorithms, with more planned:
Algorithm Quick Reference
Each algorithm has its own Jupyter Notebook (.ipynb) file with a detailed explanation and implementation.
Complementing the detailed notebooks, a comprehensive RL Cheat Sheet is included in this repository. It serves as a quick reference guide covering:
- Core RL Concepts (MDPs, Bellman Equations, etc.)
- Algorithm Summaries (Core Idea, Math, Pseudocode)
- Key Hyperparameters and Tuning Tips
- Pros & Cons and Use Cases
- Code Snippets for key update rules
β‘οΈ View the RL Cheat Sheet here
-
Clone the repository:
git clone https://github.com/fareedkhan-dev/all-rl-algorithms.git cd all-rl-algorithms -
Install the dependencies with uv, which reads
pyproject.tomlanduv.lockand creates.venv/for you:uv sync
Or with pip, in a virtual environment of your own (Python 3.10 or newer):
python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install -r requirements.txt jupyter
-
Start Jupyter and open any notebook:
uv run jupyter lab, orjupyter labwith pip. -
A3C: run the script version from a terminal,
uv run python a3c_training.py, because multiprocessing does not work reliably inside a notebook.
π‘ No uv yet? See the installation guide, or
pip install uv.
- Start with the Basics (
01_simple_rl.ipynb): This notebook introduces fundamental RL concepts like states, actions, rewards, and policies. - Explore Core Algorithms: Dive into the individual notebooks for Q-Learning (
02_q_learning.ipynb), SARSA (03_sarsa.ipynb), and REINFORCE (06_reinforce.ipynb). Understand their update rules, strengths, and limitations. - Analyze the Code: Carefully read the code comments, which explain the purpose of each function and variable.
- Experiment!: This is where the real learning begins. Try these:
- Change hyperparameters (learning rate, discount factor, exploration rate) and observe the effect on learning curves.
- Modify the environment (e.g., change the grid size, add obstacles) and see how the algorithms adapt.
- Extend the algorithms (e.g., implement epsilon decay, add a baseline to REINFORCE).
- Consult the Cheat Sheet: Refer to the RL Cheat Sheet for quick summaries, formulas, and pseudocode while studying the notebooks.
- Tackle Advanced Methods: Gradually work through the more complex notebooks on DQN (
13_dqn.ipynb), Actor-Critic (08_a2c.ipynb), PPO (07_ppo.ipynb), Model-Based RL with PlaNet (18_planet.ipynb), and multi-agent learning with MADDPG (14_maddpg.ipynb) and QMIX (15_qmix.ipynb). - Run the A3C Implementation: Due to complexities with multiprocessing in Jupyter Notebooks, the A3C implementation is in
a3c_training.py. Run it from the command line:python a3c_training.py
Each notebook includes visualizations to help you understand the agent's behavior:
- Learning Curves: Plots of episode rewards, episode lengths, and loss functions.
- Q-Table Visualizations: Heatmaps to visualize Q-values across the state space (tabular methods).
- Policy Grids: Arrows showing the learned policy (action choice) in each state.
- More Advanced Visualizations: Visualizations may depend on each particular algorithm.
This repository is for learning. Several notebooks simplify the published algorithms to keep them readable (PlaNet, HAC and MADDPG among them), and each one says where. The outputs and the analysis next to them come from running each notebook from the top on its current code. If you find a bug, please open an issue or a pull request.
Contributions are welcome! Here's how you can help:
- π Fix Bugs: Found an error or a way to improve the code? Submit a pull request!
- βοΈ Improve Explanations: Clarify confusing sections in the notebooks or add more helpful comments.
- β‘ Add More Algorithms: Implement algorithms currently marked as "Planned."
- π Create More Visualizations: Develop insightful visualizations to better understand the learning process.
- π Add More Environment Examples: Implement known RL tasks.
- π Follow Guidelines: Please follow the project's coding style and documentation guidelines.
- π Open Discussions: Create a new issue to discuss your contribution before starting work.
If you change a notebook, re-run it from the top (Restart Kernel and Run All) and commit it with its outputs, so the outputs always match the code. Then run the tests with uv run pytest: they check that every notebook ran in order without errors and that the algorithms behave as expected on small cases worked out by hand. CI runs the same tests on every pull request.