In reinforcement learning (RL) problems, learning agents sequentially execute actions with the goal of maximizing a reward signal. The RL framework has gained popularity with the development of algorithms capable of mastering increasingly complex problems, but learning difficult tasks is often slow or infeasible when RL agents begin with no prior knowledge. The key insight behind "transfer learning" is that generalization may occur not only within tasks, but also across tasks. While transfer has been studied in the psychological literature for many years, the RL community has only recently begun to investigate the benefits of transferring knowledge. This book provides an introduction to the RL transfer problem and discusses methods which demonstrate the promise of this exciting area of research. The key contributions of this book are: Definition of the transfer problem in RL domains Background on RL, sufficient to allow a wide audience to understand discussed transfer concepts Taxonomy for transfer methods in RL Survey of existing approaches In-depth presentation of selected transfer methods Discussion of key open questions By way of the research presented in this book, the author has established himself as the pre-eminent worldwide expert on transfer learning in sequential decision making tasks. A particular strength of the research is its very thorough and methodical empirical evaluation, which Matthew presents, motivates, and analyzes clearly in prose throughout the book. Whether this is your initial introduction to the concept of transfer learning, or whether you are a practitioner in the field looking for nuanced details, I trust that you will find this book to be an enjoyable and enlightening read. Peter Stone, Associate Professor of Computer Science

Reinforcement learning (RL) methods have become popular in recent years because of their ability to solve complex tasks with minimal feedback. While these methods have had experimental successes and have been shown to exhibit some desirable properties in theory, the basic learning algorithms have often been found slow in practice. Therefore, much of the current RL research focuses on speeding up learning by taking advantage of domain knowledge, or by better utilizing agents' experience. The ambitious goal of transfer learning, when applied to RL tasks, is to accelerate learning on some target task after training on a different, but related, source task. This dissertation demonstrates that transfer learning methods can successfully improve learning in RL tasks via experience from previously learned tasks. Transfer learning can increase RL's applicability to difficult tasks by allowing agents to generalize their experience across learning problems. This dissertation presents inter-task mappings, the first transfer mechanism in this area to successfully enable transfer between tasks with different state variables and actions. Inter-task mappings have subsequently been used by a number of transfer researchers. A set of six transfer learning algorithms are then introduced. While these transfer methods differ in terms of what base RL algorithms they are compatible with, what type of knowledge they transfer, and what their strengths are, all utilize the same inter-task mapping mechanism. These transfer methods can all successfully use mappings constructed by a human from domain knowledge, but there may be situations in which domain knowledge is unavailable, or insufficient, to describe how two given tasks are related. We therefore also study how inter-task mappings can be learned autonomously by leveraging existing machine learning algorithms. Our methods use classification and regression techniques to successfully discover similarities between data gathered in pairs of tasks, culminating in what is currently one of the most robust mapping-learning algorithms for RL transfer. Combining transfer methods with these similarity-learning algorithms allows us to empirically demonstrate the plausibility of autonomous transfer. We fully implement these methods in four domains (each with different salient characteristics), show that transfer can significantly improve an agent's ability to learn in each domain, and explore the limits of transfer's applicability.

In reinforcement learning (RL) problems, learning agents sequentially execute actions with the goal of maximizing a reward signal. The RL framework has gained popularity with the development of algorithms capable of mastering increasingly complex problems, but learning difficult tasks is often slow or infeasible when RL agents begin with no prior knowledge. The key insight behind "transfer learning" is that generalization may occur not only within tasks, but also across tasks. While transfer has been studied in the psychological literature for many years, the RL community has only recently begun to investigate the benefits of transferring knowledge. This book provides an introduction to the RL transfer problem and discusses methods which demonstrate the promise of this exciting area of research. The key contributions of this book are: Definition of the transfer problem in RL domains Background on RL, sufficient to allow a wide audience to understand discussed transfer concepts Taxonomy for transfer methods in RL Survey of existing approaches In-depth presentation of selected transfer methods Discussion of key open questions By way of the research presented in this book, the author has established himself as the pre-eminent worldwide expert on transfer learning in sequential decision making tasks. A particular strength of the research is its very thorough and methodical empirical evaluation, which Matthew presents, motivates, and analyzes clearly in prose throughout the book. Whether this is your initial introduction to the concept of transfer learning, or whether you are a practitioner in the field looking for nuanced details, I trust that you will find this book to be an enjoyable and enlightening read. Peter Stone, Associate Professor of Computer Science

This book constitutes the refereed proceedings of the joint conference on Machine Learning and Knowledge Discovery in Databases: ECML PKDD 2008, held in Antwerp, Belgium, in September 2008. The 100 papers presented in two volumes, together with 5 invited talks, were carefully reviewed and selected from 521 submissions. In addition to the regular papers the volume contains 14 abstracts of papers appearing in full version in the Machine Learning Journal and the Knowledge Discovery and Databases Journal of Springer. The conference intends to provide an international forum for the discussion of the latest high quality research results in all areas related to machine learning and knowledge discovery in databases. The topics addressed are application of machine learning and data mining methods to real-world problems, particularly exploratory research that describes novel learning and mining tasks and applications requiring non-standard techniques.

Reinforcement learning encompasses both a science of adaptive behavior of rational beings in uncertain environments and a computational methodology for finding optimal behaviors for challenging problems in control, optimization and adaptive behavior of intelligent agents. As a field, reinforcement learning has progressed tremendously in the past decade. The main goal of this book is to present an up-to-date series of survey articles on the main contemporary sub-fields of reinforcement learning. This includes surveys on partially observable environments, hierarchical task decompositions, relational knowledge representation and predictive state representations. Furthermore, topics such as transfer, evolutionary methods and continuous spaces in reinforcement learning are surveyed. In addition, several chapters review reinforcement learning methods in robotics, in games, and in computational neuroscience. In total seventeen different subfields are presented by mostly young experts in those areas, and together they truly represent a state-of-the-art of current reinforcement learning research. Marco Wiering works at the artificial intelligence department of the University of Groningen in the Netherlands. He has published extensively on various reinforcement learning topics. Martijn van Otterlo works in the cognitive artificial intelligence group at the Radboud University Nijmegen in The Netherlands. He has mainly focused on expressive knowledge representation in reinforcement learning settings.

Recent advances in deep machine learning hinge on a large amount of labeled data. Such heavy dependence on supervision data impedes the broader application of deep learning in more practical scenarios, where data annotation and labeling can be expensive (e.g. high-frequency trading) or even dangerous (e.g. training autonomous-driving models.) Transfer Learning (TL), equivalently referred to as knowledge transfer, is an effective strategy to confront such challenges. TL, by its definition, distills the external knowledge from relevant domains into the target learning domain, hence requiring fewer supervision resources than learning-from-scratch. TL is beneficial for learning tasks for which the supervision data is limited or even unavailable. It is also an essential property to realize Generalized Artificial Intelligence. In this thesis, we propose sample-efficient TL approaches using limited, sometimes unreliable resources. We take a deep look into the setting of Reinforcement Learning (RL) and Supervised Learning, and derive solutions for the two domains respectively. Especially, for RL, we focus on a problem setting called imitation learning, where the supervision from the environment is either non-available or scarcely provided, and the learning agent must transfer knowledge from exterior resources, such as demonstration examples of a previously trained expert, to learn a good policy. For supervised learning, we consider a distributed machine learning scheme called Federated Learning (FL), which is a more challenging scenario than traditional machine learning, since the training data is distributed and non-sharable during the learning process. Under this distributed setting, it is imperative to enable TL among distributed learning clients to reach a satisfiable generalization performance. We prove by both theoretical support and extensive experiments that our proposed algorithms can facilitate the machine learning process with knowledge transfer to achieve higher asymptotic performance, in a principled and more efficient manner than the prior arts.

Transfer learning deals with how systems can quickly adapt themselves to new situations, tasks and environments. It gives machine learning systems the ability to leverage auxiliary data and models to help solve target problems when there is only a small amount of data available. This makes such systems more reliable and robust, keeping the machine learning model faced with unforeseeable changes from deviating too much from expected performance. At an enterprise level, transfer learning allows knowledge to be reused so experience gained once can be repeatedly applied to the real world. For example, a pre-trained model that takes account of user privacy can be downloaded and adapted at the edge of a computer network. This self-contained, comprehensive reference text describes the standard algorithms and demonstrates how these are used in different transfer learning paradigms. It offers a solid grounding for newcomers as well as new insights for seasoned researchers and developers.