{"type":"conference","language":[{"iso":"eng"}],"abstract":[{"text":"Modern neural networks can easily fit their training set perfectly. Surprisingly, despite being “overfit” in this way, they tend to generalize well to future data, thereby defying the classic bias–variance trade-off of machine learning theory. Of the many possible explanations, a prevalent one is that training by stochastic gradient descent (SGD) imposes an implicit bias that leads it to learn simple functions, and these simple functions generalize well. However, the specifics of this implicit bias are not well understood.\r\nIn this work, we explore the smoothness conjecture which states that SGD is implicitly biased towards learning functions that are smooth. We propose several measures to formalize the intuitive notion of smoothness, and we conduct experiments to determine whether SGD indeed implicitly optimizes for these measures. Our findings rule out the possibility that smoothness measures based on first-order derivatives are being implicitly enforced. They are supportive, though, of the smoothness conjecture for measures based on second-order derivatives.","lang":"eng"}],"citation":{"chicago":"Volhejn, Vaclav, and Christoph Lampert. “Does SGD Implicitly Optimize for Smoothness?” In <i>42nd German Conference on Pattern Recognition</i>, 12544:246–59. LNCS. Springer, 2021. <a href=\"https://doi.org/10.1007/978-3-030-71278-5_18\">https://doi.org/10.1007/978-3-030-71278-5_18</a>.","apa":"Volhejn, V., &#38; Lampert, C. (2021). Does SGD implicitly optimize for smoothness? In <i>42nd German Conference on Pattern Recognition</i> (Vol. 12544, pp. 246–259). Tübingen, Germany: Springer. <a href=\"https://doi.org/10.1007/978-3-030-71278-5_18\">https://doi.org/10.1007/978-3-030-71278-5_18</a>","mla":"Volhejn, Vaclav, and Christoph Lampert. “Does SGD Implicitly Optimize for Smoothness?” <i>42nd German Conference on Pattern Recognition</i>, vol. 12544, Springer, 2021, pp. 246–59, doi:<a href=\"https://doi.org/10.1007/978-3-030-71278-5_18\">10.1007/978-3-030-71278-5_18</a>.","short":"V. Volhejn, C. Lampert, in:, 42nd German Conference on Pattern Recognition, Springer, 2021, pp. 246–259.","ista":"Volhejn V, Lampert C. 2021. Does SGD implicitly optimize for smoothness? 42nd German Conference on Pattern Recognition. DAGM GCPR: German Conference on Pattern Recognition LNCS vol. 12544, 246–259.","ieee":"V. Volhejn and C. Lampert, “Does SGD implicitly optimize for smoothness?,” in <i>42nd German Conference on Pattern Recognition</i>, Tübingen, Germany, 2021, vol. 12544, pp. 246–259.","ama":"Volhejn V, Lampert C. Does SGD implicitly optimize for smoothness? In: <i>42nd German Conference on Pattern Recognition</i>. Vol 12544. LNCS. Springer; 2021:246-259. doi:<a href=\"https://doi.org/10.1007/978-3-030-71278-5_18\">10.1007/978-3-030-71278-5_18</a>"},"series_title":"LNCS","oa_version":"Submitted Version","file":[{"date_updated":"2022-08-12T07:27:58Z","creator":"dernst","success":1,"access_level":"open_access","checksum":"3e3628ab1cf658d82524963f808004ea","file_name":"2020_GCPR_submitted_Volhejn.pdf","content_type":"application/pdf","relation":"main_file","file_size":420234,"file_id":"11820","date_created":"2022-08-12T07:27:58Z"}],"date_published":"2021-03-17T00:00:00Z","volume":12544,"ddc":["510"],"year":"2021","day":"17","status":"public","publisher":"Springer","publication":"42nd German Conference on Pattern Recognition","date_updated":"2022-08-12T07:28:47Z","scopus_import":"1","has_accepted_license":"1","conference":{"name":"DAGM GCPR: German Conference on Pattern Recognition ","location":"Tübingen, Germany","start_date":"2020-09-28","end_date":"2020-10-01"},"publication_identifier":{"issn":["0302-9743"],"isbn":["9783030712778"],"eissn":["1611-3349"]},"_id":"9210","author":[{"id":"d5235fb4-7a6d-11eb-b254-f25d12d631a8","last_name":"Volhejn","full_name":"Volhejn, Vaclav","first_name":"Vaclav"},{"orcid":"0000-0001-8622-7887","id":"40C20FD2-F248-11E8-B48F-1D18A9856A87","full_name":"Lampert, Christoph","last_name":"Lampert","first_name":"Christoph"}],"file_date_updated":"2022-08-12T07:27:58Z","intvolume":"     12544","quality_controlled":"1","department":[{"_id":"ChLa"}],"publication_status":"published","article_processing_charge":"No","page":"246-259","doi":"10.1007/978-3-030-71278-5_18","title":"Does SGD implicitly optimize for smoothness?","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","month":"03","oa":1,"date_created":"2021-03-01T09:01:16Z"}