MARL: Population Training with Exact PSRO

About this event

Last meeting (https://youtu.be/Tt-8F6rcHEo) we revisited the population training algorithm introduced in Chapter 9 of Multi-Agent Reinforcement Learning: Foundations and Modern Approaches (https://www.marl-book.com/) and considered only tabular problems with exact solution techniques. That version of the algorithm has some limitations regarding the ability to learn stochastic equilibrium solutions which is often required in stochastic games with simultaneous actions. In this meeting we will explain the origin of the deficiency and how the stochastic game scenario differs from the problem used in the original paper (https://www.cs.cmu.edu/~ggordon/mcmahan-ggordon-blum.icml2003.pdf) describing the double oracle algorithm. Then we will derive the method needed to apply Kuhn's theorem and build a tabular MDP from a population distribution of opponent behavior. Finally, we can then create an exact tabular version of the PSRO algorithm and demonstrate the equivalence of its equilibrium policies with that of a value iteration solution. For additional reading, see the paper introducing PSRO (https://proceedings.neurips.cc/paper_files/paper/2017/file/3323fe11e9595c09af38fe67567a9394-Paper.pdf) Meetup Links: Recordings of Previous RL Meetings (https://youtube.com/playlist?list=PLYqXmZaxvwmy2CNaK-DLailou1VIU1UZn&si=n6uQm863MCcHuKT7) Recordings of Previous MARL Meetings (https://youtube.com/playlist?list=PLYqXmZaxvwmzikjw-cNyZfI051ms05czB&si=A7-AeX0dcRW67PDB) Short RL Tutorials (https://youtube.com/playlist?list=PLYqXmZaxvwmyLEXMpk-n4RFr59tpJjNXt&si=RHy_FAnOJnPa4p1N) My exercise solutions and chapter notes for Sutton-Barto (https://github.com/jekyllstein/Reinforcement-Learning-Sutton-Barto-Exercise-Solutions) My MARL repository (https://github.com/jekyllstein/MARL_course/tree/main) Kickoff Slides which contain other links (https://docs.google.com/presentation/d/1QD3iw5BgIpPpl_K_ApAlDr1NRseR1WmXme1dKQGqTOg/edit?usp=sharing) MARL Kickoff Slides (https://docs.google.com/presentation/d/1FHXGVWkzjKsnNxzVN-29dx5vdkffAx5Vji5nWrDvg1Y/edit?usp=sharing) MARL Links: Multi-Agent Reinforcement Learning: Foundations and Modern Approaches (https://www.marl-book.com/) MARL Summer Course Videos (https://youtube.com/playlist?list=PLkoCa1tf0XjCU6GkAfRCkChOOSH6-JC_2&si=lEljXo65s3fMUsRC) MARL Slides (https://github.com/marl-book/slides) Sutton and Barto Links: Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto (http://incompleteideas.net/book/the-book.html) Video lectures from a similar course (https://youtube.com/playlist?list=PLqYmG7hTraZDVH599EItlEWsUOsJbAodm)