---
res:
  bibo_abstract:
  - 'Understanding the properties of neural networks trained via stochastic gradient
    descent (SGD) is at the heart of the theory of deep learning. In this work, we
    take a mean-field view, and consider a two-layer ReLU network trained via noisy-SGD
    for a univariate regularized regression problem. Our main result is that SGD with
    vanishingly small noise injected in the gradients is biased towards a simple solution:
    at convergence, the ReLU network implements a piecewise linear map of the inputs,
    and the number of “knot” points -- i.e., points where the tangent of the ReLU
    network estimator changes -- between two consecutive training inputs is at most
    three. In particular, as the number of neurons of the network grows, the SGD dynamics
    is captured by the solution of a gradient flow and, at convergence, the distribution
    of the weights approaches the unique minimizer of a related free energy, which
    has a Gibbs form. Our key technical contribution consists in the analysis of the
    estimator resulting from this minimizer: we show that its second derivative vanishes
    everywhere, except at some specific locations which represent the “knot” points.
    We also provide empirical evidence that knots at locations distinct from the data
    points might occur, as predicted by our theory.@eng'
  bibo_authorlist:
  - foaf_Person:
      foaf_givenName: Aleksandr
      foaf_name: Shevchenko, Aleksandr
      foaf_surname: Shevchenko
      foaf_workInfoHomepage: http://www.librecat.org/personId=F2B06EC2-C99E-11E9-89F0-752EE6697425
  - foaf_Person:
      foaf_givenName: Vyacheslav
      foaf_name: Kungurtsev, Vyacheslav
      foaf_surname: Kungurtsev
  - foaf_Person:
      foaf_givenName: Marco
      foaf_name: Mondelli, Marco
      foaf_surname: Mondelli
      foaf_workInfoHomepage: http://www.librecat.org/personId=27EB676C-8706-11E9-9510-7717E6697425
    orcid: 0000-0002-3242-7020
  bibo_issue: '130'
  bibo_volume: 23
  dct_date: 2022^xs_gYear
  dct_isPartOf:
  - http://id.crossref.org/issn/1532-4435
  - http://id.crossref.org/issn/1533-7928
  dct_language: eng
  dct_publisher: Journal of Machine Learning Research@
  dct_title: Mean-field analysis of piecewise linear solutions for wide ReLU networks@
...