---
res:
bibo_abstract:
- 'Understanding the properties of neural networks trained via stochastic gradient
descent (SGD) is at the heart of the theory of deep learning. In this work, we
take a mean-field view, and consider a two-layer ReLU network trained via noisy-SGD
for a univariate regularized regression problem. Our main result is that SGD with
vanishingly small noise injected in the gradients is biased towards a simple solution:
at convergence, the ReLU network implements a piecewise linear map of the inputs,
and the number of “knot” points -- i.e., points where the tangent of the ReLU
network estimator changes -- between two consecutive training inputs is at most
three. In particular, as the number of neurons of the network grows, the SGD dynamics
is captured by the solution of a gradient flow and, at convergence, the distribution
of the weights approaches the unique minimizer of a related free energy, which
has a Gibbs form. Our key technical contribution consists in the analysis of the
estimator resulting from this minimizer: we show that its second derivative vanishes
everywhere, except at some specific locations which represent the “knot” points.
We also provide empirical evidence that knots at locations distinct from the data
points might occur, as predicted by our theory.@eng'
bibo_authorlist:
- foaf_Person:
foaf_givenName: Aleksandr
foaf_name: Shevchenko, Aleksandr
foaf_surname: Shevchenko
foaf_workInfoHomepage: http://www.librecat.org/personId=F2B06EC2-C99E-11E9-89F0-752EE6697425
- foaf_Person:
foaf_givenName: Vyacheslav
foaf_name: Kungurtsev, Vyacheslav
foaf_surname: Kungurtsev
- foaf_Person:
foaf_givenName: Marco
foaf_name: Mondelli, Marco
foaf_surname: Mondelli
foaf_workInfoHomepage: http://www.librecat.org/personId=27EB676C-8706-11E9-9510-7717E6697425
orcid: 0000-0002-3242-7020
bibo_issue: '130'
bibo_volume: 23
dct_date: 2022^xs_gYear
dct_isPartOf:
- http://id.crossref.org/issn/1532-4435
- http://id.crossref.org/issn/1533-7928
dct_language: eng
dct_publisher: Journal of Machine Learning Research@
dct_title: Mean-field analysis of piecewise linear solutions for wide ReLU networks@
...