Title: Causally Correct Partial Models for Reinforcement Learning

Authors: Danilo J. Rezende, Ivo Danihelka, George Papamakarios, Nan Rosemary Ke, Ray Jiang, Theophane Weber, Karol Gregor, Hamza Merzic, Fabio Viola, Jane Wang, Jovana Mitrovic, Frederic Besse, Ioannis Antonoglou, Lars Buesing

Abstract: In reinforcement learning, we can learn a model of future observations and rewards, and use it to plan the agent’s next actions. However, jointly modeling future observations can be computation- ally expensive or even intractable if the observa- tions are high-dimensional (e.g. images). For this reason, previous works have considered partial models, which model only part of the observa- tion. In this paper, we show that partial models can be causally incorrect: they are confounded by the observations they don’t model, and can there- fore lead to incorrect planning. To address this, we introduce a general family of partial models that are provably causally correct, yet remain fast because they do not need to fully model future observations.

link