{"department":[{"_id":"ToHe"}],"oa_version":"None","language":[{"iso":"eng"}],"type":"conference","abstract":[{"text":"There is an ever-growing zoo of modern neural network models that can efficiently learn end-to-end control from visual observations. These advanced deep models, ranging from convolutional to Vision Transformers, from small to gigantic networks, have been extensively tested on offline image classification tasks. In this paper, we study these vision models with respect to the open-loop training to closed-loop generalization abilities, i.e., deployment realizes a causal feedback loop that is not present during training. This causality gap typically emerges in robotics applications such as autonomous driving, where a network is trained to imitate the control commands of a human. In this setting, two situations arise: 1) Closed-loop testing in-distribution, where the test environment shares properties with those of offline training data. 2) Closed-loop testing under distribution shifts and out-of-distribution. Contrary to recently reported results, we show that under proper training guidelines, all vision architectures perform indistinguishably well on in-distribution deployment, resolving the causality gap. In situation 2, We observe that scale is the strongest factor in improving closed-loop generalization regardless of the choice of the model architecture. Our results predict the trend that in the future we will see larger and larger models being used in offline-training-online-deployment imitation learning tasks in robotic applications.","lang":"eng"}],"ec_funded":1,"month":"08","_id":"17898","title":"Overparametrization helps offline-to-online generalization of closed-loop control from pixels","citation":{"mla":"Lechner, Mathias, et al. “Overparametrization Helps Offline-to-Online Generalization of Closed-Loop Control from Pixels.” <i>Proceedings of the 2024 IEEE International Conference on Robotics and Automation</i>, Institute of Electrical and Electronics Engineers, 2024, pp. 2774–82, doi:<a href=\"https://doi.org/10.1109/ICRA57147.2024.10610284\">10.1109/ICRA57147.2024.10610284</a>.","ama":"Lechner M, Hasani R, Amini A, Wang TH, Henzinger TA, Rus D. Overparametrization helps offline-to-online generalization of closed-loop control from pixels. In: <i>Proceedings of the 2024 IEEE International Conference on Robotics and Automation</i>. Institute of Electrical and Electronics Engineers; 2024:2774-2782. doi:<a href=\"https://doi.org/10.1109/ICRA57147.2024.10610284\">10.1109/ICRA57147.2024.10610284</a>","chicago":"Lechner, Mathias, Ramin Hasani, Alexander Amini, Tsun Hsuan Wang, Thomas A Henzinger, and Daniela Rus. “Overparametrization Helps Offline-to-Online Generalization of Closed-Loop Control from Pixels.” In <i>Proceedings of the 2024 IEEE International Conference on Robotics and Automation</i>, 2774–82. Institute of Electrical and Electronics Engineers, 2024. <a href=\"https://doi.org/10.1109/ICRA57147.2024.10610284\">https://doi.org/10.1109/ICRA57147.2024.10610284</a>.","ieee":"M. Lechner, R. Hasani, A. Amini, T. H. Wang, T. A. Henzinger, and D. Rus, “Overparametrization helps offline-to-online generalization of closed-loop control from pixels,” in <i>Proceedings of the 2024 IEEE International Conference on Robotics and Automation</i>, Yokohama, Japan, 2024, pp. 2774–2782.","ista":"Lechner M, Hasani R, Amini A, Wang TH, Henzinger TA, Rus D. 2024. Overparametrization helps offline-to-online generalization of closed-loop control from pixels. Proceedings of the 2024 IEEE International Conference on Robotics and Automation. ICRA: Conference on Robotics and Automation, 2774–2782.","apa":"Lechner, M., Hasani, R., Amini, A., Wang, T. H., Henzinger, T. A., &#38; Rus, D. (2024). Overparametrization helps offline-to-online generalization of closed-loop control from pixels. In <i>Proceedings of the 2024 IEEE International Conference on Robotics and Automation</i> (pp. 2774–2782). Yokohama, Japan: Institute of Electrical and Electronics Engineers. <a href=\"https://doi.org/10.1109/ICRA57147.2024.10610284\">https://doi.org/10.1109/ICRA57147.2024.10610284</a>","short":"M. Lechner, R. Hasani, A. Amini, T.H. Wang, T.A. Henzinger, D. Rus, in:, Proceedings of the 2024 IEEE International Conference on Robotics and Automation, Institute of Electrical and Electronics Engineers, 2024, pp. 2774–2782."},"date_updated":"2024-09-11T05:56:25Z","quality_controlled":"1","publication_identifier":{"isbn":["9798350384574"],"issn":["1050-4729"]},"scopus_import":"1","date_published":"2024-08-08T00:00:00Z","article_processing_charge":"No","acknowledgement":"This work was partially supported in parts by the ERC-2020-AdG 101020093. Additionally, it was partially sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. This work was further supported by The Boeing Company and the Office of Naval Research (ONR) Grant N00014-18-1-2830.","conference":{"end_date":"2024-05-17","location":"Yokohama, Japan","name":"ICRA: Conference on Robotics and Automation","start_date":"2024-05-13"},"date_created":"2024-09-08T22:01:13Z","author":[{"first_name":"Mathias","id":"3DC22916-F248-11E8-B48F-1D18A9856A87","last_name":"Lechner","full_name":"Lechner, Mathias"},{"first_name":"Ramin","last_name":"Hasani","full_name":"Hasani, Ramin"},{"first_name":"Alexander","last_name":"Amini","full_name":"Amini, Alexander"},{"first_name":"Tsun Hsuan","last_name":"Wang","full_name":"Wang, Tsun Hsuan"},{"first_name":"Thomas A","id":"40876CD8-F248-11E8-B48F-1D18A9856A87","last_name":"Henzinger","orcid":"0000-0002-2985-7724","full_name":"Henzinger, Thomas A"},{"full_name":"Rus, Daniela","last_name":"Rus","first_name":"Daniela"}],"year":"2024","day":"08","doi":"10.1109/ICRA57147.2024.10610284","publication":"Proceedings of the 2024 IEEE International Conference on Robotics and Automation","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","publication_status":"published","project":[{"grant_number":"101020093","_id":"62781420-2b32-11ec-9570-8d9b63373d4d","call_identifier":"H2020","name":"Vigilant Algorithmic Monitoring of Software"}],"publisher":"Institute of Electrical and Electronics Engineers","status":"public","page":"2774-2782"}