upb-lea/reinforcement_learning_course_materials
Jupyter NotebookLecture notes, tutorial tasks including solutions as well as online videos for the reinforcement learning course hosted by Paderborn University
Reinforcement learning course
This work is licensed under a Creative Commons Attribution 4.0 International Public License.
Lecture notes, tutorial tasks including solutions as well as online videos for a reinforcement learning course originally hosted at Paderborn University and transferred to University of Siegen. Source code for the entire course material is open and everyone is cordially invited to use it for self-learning (students) or to set up your own course (lecturers).
Lecture slides (click on preview picture)
Introduction to reinforcement learning
Markov decision processes
Dynamic programming
Monte Carlo methods
Temporal-difference learning
Multi-step bootstrapping
Planning and learning with tabular methods
Function approximation with supervised learning
On-policy prediction with function approximation
Value-based control with function approximation
Stochastic policy gradient methods
Deterministic policy gradient methods
Further contemporary RL algorithms (TRPO, PPO)
Outlook and Research Insights
Summary of part one: reinforcement learning in finite state and action spaces
Summary of part two: reinforcement learning in continuous state and action spaces
Exercise content
All exercises are based on Python 3.12 and site-packages according to the requirements.txt:
>>> pip install -r requirements.txt
- Basics of Python for scientific computing
- Tutorial video (only 2022 edition available due to technical outage)
- Tutorial template
- Tutorial solution
- Manually solving basic Markov chain, reward and decision problems
- The beer-bachelor and dynamic programming (the shortest beer problem)
- Drive through the race track with Monte Carlo learning
- Drive even faster using temporal-difference learning
- Stabilizing the inverted pendulum by tabular multi-step methods
- Boosting the inverted pendulum by integrating learning & planning (Dyna framework)
- Predicting the operating behavior of a real electric drive systems with supervised learning
- Evaluate the performance of given agents in the mountain car problem using function approximation
- Escape from the mountain car valley using semi-gradient SARSA & least square policy iteration
- Landing on the moon with REINFORCE and actor-critic methods
- Shoot for the moon with DDPG & PPO
Contributions
We highly appreciate any feedback and input to the course material, for example:
- Reporting typos or content-related issues (please open an issue)
- Suggesting improvements or corrections (also via issues)
- Adding or altering content (please submit a pull request)
If you would like to contribute in a larger capacity (e.g., developing new lectures or exercises, maintaining content, or collaborating on course design), please open an issue first so we can coordinate.
Credits
The lecture notes are inspired by
- Richard S. Sutton, Andrew G. Barto, 'Reinforcement Learning: An Introduction' Second Edition MIT Press, Cambridge, MA, 2018
- David Silver, UCL Course on Reinforcement Learning, 2015
The tutorials are partly using pre-packed environments from
- Gymnasium (maintained branch of OpenAI's Gym)