The role of pretrained representations for the OOD generalization of reinforcement learning agents

conference paper The role of pretrained representations for the OOD generalization of reinforcement learning agents published yes Andrea Dittadi author Frederik Träuble author Manuel Wüthrich author Felix Widmaier author Peter Gehler author Ole Winther author Francesco Locatello author 26cfd52f-2483-11ee-8040-88983bcc06d40000-0002-4850-0683 Olivier Bachem author Bernhard Schölkopf author Stefan Bauer author FrLo department ICLR: International Conference on Learning Representations Building sample-efficient agents that generalize out-of-distribution (OOD) in real-world settings remains a fundamental unsolved problem on the path towards achieving higher-level cognition. One particularly promising approach is to begin with low-dimensional, pretrained representations of our world, which should facilitate efficient downstream learning and generalization. By training 240 representations and over 10,000 reinforcement learning (RL) policies on a simulated robotic setup, we evaluate to what extent different properties of pretrained VAE-based representations affect the OOD generalization of downstream agents. We observe that many agents are surprisingly robust to realistic distribution shifts, including the challenging sim-to-real case. In addition, we find that the generalization performance of a simple downstream proxy task reliably predicts the generalization performance of our RL agents under a wide range of OOD settings. Such proxy tasks can thus be used to select pretrained representations that will lead to agents that generalize. 2022Virtual eng 10th International Conference on Learning Representations 2107.05686 yes Dittadi A, Träuble F, Wüthrich M, Widmaier F, Gehler P, Winther O, Locatello F, Bachem O, Schölkopf B, Bauer S. 2022. The role of pretrained representations for the OOD generalization of reinforcement learning agents. 10th International Conference on Learning Representations. ICLR: International Conference on Learning Representations. Dittadi, Andrea, et al. “The Role of Pretrained Representations for the OOD Generalization of Reinforcement Learning Agents.” 10th International Conference on Learning Representations, 2022. Dittadi, A., Träuble, F., Wüthrich, M., Widmaier, F., Gehler, P., Winther, O., … Bauer, S. (2022). The role of pretrained representations for the OOD generalization of reinforcement learning agents. In 10th International Conference on Learning Representations. Virtual. A. Dittadi et al., “The role of pretrained representations for the OOD generalization of reinforcement learning agents,” in 10th International Conference on Learning Representations, Virtual, 2022. Dittadi A, Träuble F, Wüthrich M, et al. The role of pretrained representations for the OOD generalization of reinforcement learning agents. In: 10th International Conference on Learning Representations. ; 2022. Dittadi, Andrea, Frederik Träuble, Manuel Wüthrich, Felix Widmaier, Peter Gehler, Ole Winther, Francesco Locatello, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. “The Role of Pretrained Representations for the OOD Generalization of Reinforcement Learning Agents.” In 10th International Conference on Learning Representations, 2022. A. Dittadi, F. Träuble, M. Wüthrich, F. Widmaier, P. Gehler, O. Winther, F. Locatello, O. Bachem, B. Schölkopf, S. Bauer, in:, 10th International Conference on Learning Representations, 2022. 141742023-08-22T14:02:13Z2023-09-11T09:48:36Z