---
_id: '17898'
abstract:
- lang: eng
  text: 'There is an ever-growing zoo of modern neural network models that can efficiently
    learn end-to-end control from visual observations. These advanced deep models,
    ranging from convolutional to Vision Transformers, from small to gigantic networks,
    have been extensively tested on offline image classification tasks. In this paper,
    we study these vision models with respect to the open-loop training to closed-loop
    generalization abilities, i.e., deployment realizes a causal feedback loop that
    is not present during training. This causality gap typically emerges in robotics
    applications such as autonomous driving, where a network is trained to imitate
    the control commands of a human. In this setting, two situations arise: 1) Closed-loop
    testing in-distribution, where the test environment shares properties with those
    of offline training data. 2) Closed-loop testing under distribution shifts and
    out-of-distribution. Contrary to recently reported results, we show that under
    proper training guidelines, all vision architectures perform indistinguishably
    well on in-distribution deployment, resolving the causality gap. In situation
    2, We observe that scale is the strongest factor in improving closed-loop generalization
    regardless of the choice of the model architecture. Our results predict the trend
    that in the future we will see larger and larger models being used in offline-training-online-deployment
    imitation learning tasks in robotic applications.'
acknowledgement: This work was partially supported in parts by the ERC-2020-AdG 101020093.
  Additionally, it was partially sponsored by the United States Air Force Research
  Laboratory and the United States Air Force Artificial Intelligence Accelerator and
  was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views
  and conclusions contained in this document are those of the authors and should not
  be interpreted as representing the official policies, either expressed or implied,
  of the United States Air Force or the U.S. Government. The U.S. Government is authorized
  to reproduce and distribute reprints for Government purposes notwithstanding any
  copyright notation herein. This work was further supported by The Boeing Company
  and the Office of Naval Research (ONR) Grant N00014-18-1-2830.
article_processing_charge: No
author:
- first_name: Mathias
  full_name: Lechner, Mathias
  id: 3DC22916-F248-11E8-B48F-1D18A9856A87
  last_name: Lechner
- first_name: Ramin
  full_name: Hasani, Ramin
  last_name: Hasani
- first_name: Alexander
  full_name: Amini, Alexander
  last_name: Amini
- first_name: Tsun Hsuan
  full_name: Wang, Tsun Hsuan
  last_name: Wang
- first_name: Thomas A
  full_name: Henzinger, Thomas A
  id: 40876CD8-F248-11E8-B48F-1D18A9856A87
  last_name: Henzinger
  orcid: 0000-0002-2985-7724
- first_name: Daniela
  full_name: Rus, Daniela
  last_name: Rus
citation:
  ama: 'Lechner M, Hasani R, Amini A, Wang TH, Henzinger TA, Rus D. Overparametrization
    helps offline-to-online generalization of closed-loop control from pixels. In:
    <i>Proceedings of the 2024 IEEE International Conference on Robotics and Automation</i>.
    Institute of Electrical and Electronics Engineers; 2024:2774-2782. doi:<a href="https://doi.org/10.1109/ICRA57147.2024.10610284">10.1109/ICRA57147.2024.10610284</a>'
  apa: 'Lechner, M., Hasani, R., Amini, A., Wang, T. H., Henzinger, T. A., &#38; Rus,
    D. (2024). Overparametrization helps offline-to-online generalization of closed-loop
    control from pixels. In <i>Proceedings of the 2024 IEEE International Conference
    on Robotics and Automation</i> (pp. 2774–2782). Yokohama, Japan: Institute of
    Electrical and Electronics Engineers. <a href="https://doi.org/10.1109/ICRA57147.2024.10610284">https://doi.org/10.1109/ICRA57147.2024.10610284</a>'
  chicago: Lechner, Mathias, Ramin Hasani, Alexander Amini, Tsun Hsuan Wang, Thomas
    A Henzinger, and Daniela Rus. “Overparametrization Helps Offline-to-Online Generalization
    of Closed-Loop Control from Pixels.” In <i>Proceedings of the 2024 IEEE International
    Conference on Robotics and Automation</i>, 2774–82. Institute of Electrical and
    Electronics Engineers, 2024. <a href="https://doi.org/10.1109/ICRA57147.2024.10610284">https://doi.org/10.1109/ICRA57147.2024.10610284</a>.
  ieee: M. Lechner, R. Hasani, A. Amini, T. H. Wang, T. A. Henzinger, and D. Rus,
    “Overparametrization helps offline-to-online generalization of closed-loop control
    from pixels,” in <i>Proceedings of the 2024 IEEE International Conference on Robotics
    and Automation</i>, Yokohama, Japan, 2024, pp. 2774–2782.
  ista: 'Lechner M, Hasani R, Amini A, Wang TH, Henzinger TA, Rus D. 2024. Overparametrization
    helps offline-to-online generalization of closed-loop control from pixels. Proceedings
    of the 2024 IEEE International Conference on Robotics and Automation. ICRA: International
    Conference on Robotics and Automation, 2774–2782.'
  mla: Lechner, Mathias, et al. “Overparametrization Helps Offline-to-Online Generalization
    of Closed-Loop Control from Pixels.” <i>Proceedings of the 2024 IEEE International
    Conference on Robotics and Automation</i>, Institute of Electrical and Electronics
    Engineers, 2024, pp. 2774–82, doi:<a href="https://doi.org/10.1109/ICRA57147.2024.10610284">10.1109/ICRA57147.2024.10610284</a>.
  short: M. Lechner, R. Hasani, A. Amini, T.H. Wang, T.A. Henzinger, D. Rus, in:,
    Proceedings of the 2024 IEEE International Conference on Robotics and Automation,
    Institute of Electrical and Electronics Engineers, 2024, pp. 2774–2782.
conference:
  end_date: 2024-05-17
  location: Yokohama, Japan
  name: 'ICRA: International Conference on Robotics and Automation'
  start_date: 2024-05-13
date_created: 2024-09-08T22:01:13Z
date_published: 2024-08-08T00:00:00Z
date_updated: 2025-09-08T09:16:28Z
day: '08'
department:
- _id: ToHe
doi: 10.1109/ICRA57147.2024.10610284
ec_funded: 1
external_id:
  isi:
  - '001294576202044'
isi: 1
language:
- iso: eng
month: '08'
oa_version: None
page: 2774-2782
project:
- _id: 62781420-2b32-11ec-9570-8d9b63373d4d
  call_identifier: H2020
  grant_number: '101020093'
  name: Vigilant Algorithmic Monitoring of Software
publication: Proceedings of the 2024 IEEE International Conference on Robotics and
  Automation
publication_identifier:
  isbn:
  - '9798350384574'
  issn:
  - 1050-4729
publication_status: published
publisher: Institute of Electrical and Electronics Engineers
quality_controlled: '1'
scopus_import: '1'
status: public
title: Overparametrization helps offline-to-online generalization of closed-loop control
  from pixels
type: conference
user_id: 317138e5-6ab7-11ef-aa6d-ffef3953e345
year: '2024'
...