The Optimal Reward Problem: Designing Effective Reward for Bounded Agents.

[摘要] In the field of reinforcement learning, agent designers build agents which seek to maximize reward. In standard practice, one reward function serves two purposes. It is used to evaluate the agent and is used to directly guide agent behavior in the agent;;s learning algorithm. This dissertation makes four main contributions to the theory and practice of reward function design. The first is a demonstration that if an agent is bounded---if it is limited in its ability to maximize expected reward---the designer may benefit by considering two reward functions. A designer reward function is used to evaluate the agent, while a separate agent reward function is used to guide agent behavior. The designer can then solve the Optimal Reward Problem (ORP): choose the agent reward function which leads to the greatest expected reward for the designer.The second contribution is the demonstration through examples that good reward functions are chosen by assessing an agent;;s limitations and how they interact with the environment. An agent which maintains knowledge of the environment in the form of a Bayesian posterior distribution, but lacks adequate planning resources, can be given a reward proportional to the variance of the posterior, resulting in provably efficient exploration. An agent with poor modeling assumptions can be punished for visiting the areas of the state space it has trouble modeling, resulting in better performance.The third contribution is the Policy Gradient for Reward Design (PGRD) algorithm, a convergent gradient ascent algorithm for learning good reward functions. Experiments in multiple environments demonstrate that using PGRD for reward optimization yields better agents than using the designer;;s reward directly as the agent;;s reward. It also outperforms the use of an evaluation function at the leaf-states of the planning tree. Finally, this dissertation shows that the ORP differs from the popular work on potential-based reward shaping. Shaping rewards are constrained by properties of the environment and the designer;;s reward function, but they generally are defined irrespective of properties of the agent. The best shaping reward functions are suboptimal for some agents and environments.

[发布机构] University of Michigan

[效力级别] Computer Science

[关键词] Reinforcement Learning;Computer Science;Engineering;Computer Science & Engineering