MARL Chapter 9.9: Population-Based Training

About this event

This meeting will continue the material from Chapter 9 in Multi-Agent Reinforcement Learning: Foundations and Modern Approaches (https://www.marl-book.com/). In the last meeting we explored how policy self-play can generate generations of agents that hopefully improve in their performance. That technique was confined to zero-sum two player games and a particular type of search enhanced policy training. Today, we will extend that concept to arbitrary stochastic games and generic reinforcement learning training. The algorithm we will introduce - Policy Space Response Oracles - is designed to manage populations of agents in a stochastic game while also building a corresponding "metagame." The metagame is based on the performance of agents when they interact and selecting which generation of agent should be used for each player. We will use single-agent RL techniques to train best response policies in the "metagame" with the goal of converging to an equilibrium solution. As usual you can find below links to the textbook, previous chapter notes, slides, and recordings of some of the previous meetings. Meetup Links: Recordings of Previous RL Meetings (https://youtube.com/playlist?list=PLYqXmZaxvwmy2CNaK-DLailou1VIU1UZn&si=n6uQm863MCcHuKT7) Recordings of Previous MARL Meetings (https://youtube.com/playlist?list=PLYqXmZaxvwmzikjw-cNyZfI051ms05czB&si=A7-AeX0dcRW67PDB) Short RL Tutorials (https://youtube.com/playlist?list=PLYqXmZaxvwmyLEXMpk-n4RFr59tpJjNXt&si=RHy_FAnOJnPa4p1N) My exercise solutions and chapter notes for Sutton-Barto (https://github.com/jekyllstein/Reinforcement-Learning-Sutton-Barto-Exercise-Solutions) My MARL repository (https://github.com/jekyllstein/MARL_course/tree/main) Kickoff Slides which contain other links (https://docs.google.com/presentation/d/1QD3iw5BgIpPpl_K_ApAlDr1NRseR1WmXme1dKQGqTOg/edit?usp=sharing) MARL Kickoff Slides (https://docs.google.com/presentation/d/1FHXGVWkzjKsnNxzVN-29dx5vdkffAx5Vji5nWrDvg1Y/edit?usp=sharing) MARL Links: Multi-Agent Reinforcement Learning: Foundations and Modern Approaches (https://www.marl-book.com/) MARL Summer Course Videos (https://youtube.com/playlist?list=PLkoCa1tf0XjCU6GkAfRCkChOOSH6-JC_2&si=lEljXo65s3fMUsRC) MARL Slides (https://github.com/marl-book/slides) Sutton and Barto Links: Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto (http://incompleteideas.net/book/the-book.html) Video lectures from a similar course (https://youtube.com/playlist?list=PLqYmG7hTraZDVH599EItlEWsUOsJbAodm)