Beyond Trial & Error: A Tutorial on Automated Reinforcement Learning

Date: 10.09.2024, 15:30-17:00

Room: 26-25/105

Speakers

•   Theresa Eimer, Leibniz University Hannover
•   André Biedenkapp, Albert-Ludwigs University Freiburg

Motivation

AutoML as a field continues to predominately focus on supervised learning as a target domain even though other fields within machine learning offer potential impact that is just as large. This tutorial serves as an introduction to one such field, Automated Reinforcement Learning (AutoRL). As Reinforcement Learning (RL) requires a significant amount of configuration and selection of algorithms, hyperparameters, and task sequencing, all of which are currently predominantly done by hand, AutoML methods can contribute to making RL algorithms more efficient and easier to apply. We will introduce the specific challenges that come with RL as a target domain, focusing on the dynamic configuration ideas that are especially important in this inherently nonstationary setting. Furthermore, we will discuss the current state of the art in different AutoRL categories like HPO, meta-learning, or task design to
enable AutoML researchers to apply their ideas in AutoRL.

Applying RL to a novel problem set is challenging for many different reasons, including choosing the best approach from a wealth of available algorithms, (meta)algorithmic extensions, and low level design decisions like hyperparameters and task sequencing. So far, this has been done largely by hand, leading to RL papers having significant cost overheads for, e.g., hyperparameter search. This cost, combined with the poor experimental and reporting practice often associated with it, is a significant barrier of entry to RL research and the application of RL to real-world problems. The AutoML community has significant expertise that could help solve these issues, but standard AutoML methods do not capture the unique challenges of the RL setting, such as the dynamic nature of the problem. This tutorial will enable AutoML researchers to apply their ideas to RL by introducing them to the most prominent problem settings and state-of-the-art AutoRL methods, opening an avenue for AutoML to impact a thriving research field.

The tutorial will follow our survey article on AutoRL (by Parker-Holder et al., 2022), including topics such as (dynamic) configuration or environment design that have become increasingly important in the RL community. In addition, we will extend this with recent progress in hyperparameter optimization for different RL paradigms (e.g. offline or multi-objective RL) and how AutoRL interacts with experimental design in RL.

Outline

Part 1: Introduction and algorithmic part on AutoRL

1. Motivation: Why does AutoRL matter?
2. Formal definition of AutoRL
3. Categories of AutoRL approaches (e.g. learning to learn, environment design, etc.)
4. Properties of AutoRL landscapes:
5. What are AutoRL-specific challenges compared to AutoML for supervised learning?
6. Why are dynamic configuration approaches important for RL, and how do we learn them?

Part 2: Practical guidelines and case study of hyperparameters

1. Examples of successful AutoRL, DAC and online approaches
2. Evaluation and Generalization of AutoRL
3. HPO for RL
4. Hyperparameters and experimental design
5. Forms of optimization with pros and cons (AC methods, PBT, heuristics, meta-gradients, etc.)
6. Combining HPO with other AutoRL domains and why this is important for RL generally

Speakers