---
_id: '9198'
abstract:
- lang: eng
  text: "The optimization of multilayer neural networks typically leads to a solution\r\nwith
    zero training error, yet the landscape can exhibit spurious local minima\r\nand
    the minima can be disconnected. In this paper, we shed light on this\r\nphenomenon:
    we show that the combination of stochastic gradient descent (SGD)\r\nand over-parameterization
    makes the landscape of multilayer neural networks\r\napproximately connected and
    thus more favorable to optimization. More\r\nspecifically, we prove that SGD solutions
    are connected via a piecewise linear\r\npath, and the increase in loss along this
    path vanishes as the number of\r\nneurons grows large. This result is a consequence
    of the fact that the\r\nparameters found by SGD are increasingly dropout stable
    as the network becomes\r\nwider. We show that, if we remove part of the neurons
    (and suitably rescale the\r\nremaining ones), the change in loss is independent
    of the total number of\r\nneurons, and it depends only on how many neurons are
    left. Our results exhibit\r\na mild dependence on the input dimension: they are
    dimension-free for two-layer\r\nnetworks and depend linearly on the dimension
    for multilayer networks. We\r\nvalidate our theoretical findings with numerical
    experiments for different\r\narchitectures and classification tasks."
acknowledgement: M. Mondelli was partially supported by the 2019 LopezLoreta Prize.
  The authors thank Phan-Minh Nguyen for helpful discussions and the IST Distributed
  Algorithms and Systems Lab for providing computational resources.
article_processing_charge: No
arxiv: 1
author:
- first_name: Aleksandr
  full_name: Shevchenko, Aleksandr
  id: F2B06EC2-C99E-11E9-89F0-752EE6697425
  last_name: Shevchenko
- first_name: Marco
  full_name: Mondelli, Marco
  id: 27EB676C-8706-11E9-9510-7717E6697425
  last_name: Mondelli
  orcid: 0000-0002-3242-7020
citation:
  ama: 'Shevchenko A, Mondelli M. Landscape connectivity and dropout stability of
    SGD solutions for over-parameterized neural networks. In: <i>Proceedings of the
    37th International Conference on Machine Learning</i>. Vol 119. ML Research Press;
    2020:8773-8784.'
  apa: Shevchenko, A., &#38; Mondelli, M. (2020). Landscape connectivity and dropout
    stability of SGD solutions for over-parameterized neural networks. In <i>Proceedings
    of the 37th International Conference on Machine Learning</i> (Vol. 119, pp. 8773–8784).
    ML Research Press.
  chicago: Shevchenko, Aleksandr, and Marco Mondelli. “Landscape Connectivity and
    Dropout Stability of SGD Solutions for Over-Parameterized Neural Networks.” In
    <i>Proceedings of the 37th International Conference on Machine Learning</i>, 119:8773–84.
    ML Research Press, 2020.
  ieee: A. Shevchenko and M. Mondelli, “Landscape connectivity and dropout stability
    of SGD solutions for over-parameterized neural networks,” in <i>Proceedings of
    the 37th International Conference on Machine Learning</i>, 2020, vol. 119, pp.
    8773–8784.
  ista: Shevchenko A, Mondelli M. 2020. Landscape connectivity and dropout stability
    of SGD solutions for over-parameterized neural networks. Proceedings of the 37th
    International Conference on Machine Learning. vol. 119, 8773–8784.
  mla: Shevchenko, Aleksandr, and Marco Mondelli. “Landscape Connectivity and Dropout
    Stability of SGD Solutions for Over-Parameterized Neural Networks.” <i>Proceedings
    of the 37th International Conference on Machine Learning</i>, vol. 119, ML Research
    Press, 2020, pp. 8773–84.
  short: A. Shevchenko, M. Mondelli, in:, Proceedings of the 37th International Conference
    on Machine Learning, ML Research Press, 2020, pp. 8773–8784.
date_created: 2021-02-25T09:36:22Z
date_published: 2020-07-13T00:00:00Z
date_updated: 2026-10-04T22:31:21Z
day: '13'
ddc:
- '000'
department:
- _id: MaMo
- _id: DaAl
external_id:
  arxiv:
  - '1912.10095'
file:
- access_level: open_access
  checksum: f042c8d4316bd87c6361aa76f1fbdbbe
  content_type: application/pdf
  creator: dernst
  date_created: 2021-03-02T15:38:14Z
  date_updated: 2021-03-02T15:38:14Z
  file_id: '9217'
  file_name: 2020_PMLR_Shevchenko.pdf
  file_size: 5336380
  relation: main_file
  success: 1
file_date_updated: 2021-03-02T15:38:14Z
has_accepted_license: '1'
intvolume: '       119'
language:
- iso: eng
month: '07'
oa: 1
oa_version: Published Version
page: 8773-8784
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
  name: Prix Lopez-Loretta 2019 - Marco Mondelli
publication: Proceedings of the 37th International Conference on Machine Learning
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
related_material:
  record:
  - id: '17465'
    relation: dissertation_contains
    status: public
status: public
title: Landscape connectivity and dropout stability of SGD solutions for over-parameterized
  neural networks
type: conference
user_id: 8b945eb4-e2f2-11eb-945a-df72226e66a9
volume: 119
year: '2020'
...