Improving reinforcement learning with ensembles of different learners
Date
2021
Authors
Crafford, Gerrie
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Different reinforcement learning methods exist to address the problem of combining multiple dif ferent learners to generate a superior learner, from ensemble methods to policy reuse methods.
These methods usually assume that each learner uses the same algorithm and/or state represen tation and often require learners to be pre-trained. This assumption prevents very different types
of learners, that can potentially complement each other well, from being used together.
We propose a novel algorithm, Adaptive Probabilistic Ensemble Learning (APEL), which is an
ensemble learner that combines a set of base reinforcement learners and leverages the strengths
of the different base learners online, while remaining agnostic to the inner workings of the base
learners, thereby allowing it to combine very different types of learners. The ensemble learner
selects the base learners that perform best on average by keeping track of the performance of the
base learners and then probabilistically selecting a base learner for each episode according the
historical performance of the base learners. Along with a description of the proposed algorithm,
we present a theoretical analysis of its behaviour and performance.
We demonstrate the proposed ensemble learner’s ability to select the best base learner on av erage, combine the strengths of multiple base learners, including Q-learning, deep Q-network
(DQN), Actor-Critic with Experience Replay (ACER), and learners with different state repre sentations, as well as its ability to adapt to changes in base learner performance on grid world
navigation tasks, the Cartpole domain, and the Atari Breakout domain. The effect that the en semble learner’s hyperparameter has on its behaviour and performance is also quantified through
different experiments.
Description
A dissertation submitted in partial fulfilment for the degree of Master of Science to the Faculty of Science, School of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg, 2021
Keywords
Citation
Crafford, Gerrie (2021) Improving reinforcement learning with ensembles of different learners, University of the Witwatersrand, Johannesburg, <http://hdl.handle.net/10539/32383>