{"department":[{"_id":"ChLa"}],"oa":1,"date_created":"2019-02-14T14:51:57Z","page":"2815-2824","language":[{"iso":"eng"}],"article_processing_charge":"No","month":"02","year":"2018","scopus_import":"1","isi":1,"date_published":"2018-02-01T00:00:00Z","_id":"6011","intvolume":"        80","type":"conference","author":[{"full_name":"Kuzborskij, Ilja","first_name":"Ilja","last_name":"Kuzborskij"},{"last_name":"Lampert","orcid":"0000-0001-8622-7887","first_name":"Christoph","id":"40C20FD2-F248-11E8-B48F-1D18A9856A87","full_name":"Lampert, Christoph"}],"oa_version":"Preprint","external_id":{"isi":["000683379202095"],"arxiv":["1703.01678"]},"publication_status":"published","ec_funded":1,"user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","status":"public","day":"01","main_file_link":[{"url":"https://arxiv.org/abs/1703.01678","open_access":"1"}],"title":"Data-dependent stability of stochastic gradient descent","date_updated":"2023-10-17T09:51:13Z","quality_controlled":"1","project":[{"_id":"2532554C-B435-11E9-9278-68D0E5697425","call_identifier":"FP7","name":"Lifelong Learning of Visual Scene Understanding","grant_number":"308036"}],"citation":{"mla":"Kuzborskij, Ilja, and Christoph Lampert. “Data-Dependent Stability of Stochastic Gradient Descent.” <i>Proceedings of the 35 Th International Conference on Machine Learning</i>, vol. 80, ML Research Press, 2018, pp. 2815–24.","apa":"Kuzborskij, I., &#38; Lampert, C. (2018). Data-dependent stability of stochastic gradient descent. In <i>Proceedings of the 35 th International Conference on Machine Learning</i> (Vol. 80, pp. 2815–2824). Stockholm, Sweden: ML Research Press.","short":"I. Kuzborskij, C. Lampert, in:, Proceedings of the 35 Th International Conference on Machine Learning, ML Research Press, 2018, pp. 2815–2824.","chicago":"Kuzborskij, Ilja, and Christoph Lampert. “Data-Dependent Stability of Stochastic Gradient Descent.” In <i>Proceedings of the 35 Th International Conference on Machine Learning</i>, 80:2815–24. ML Research Press, 2018.","ieee":"I. Kuzborskij and C. Lampert, “Data-dependent stability of stochastic gradient descent,” in <i>Proceedings of the 35 th International Conference on Machine Learning</i>, Stockholm, Sweden, 2018, vol. 80, pp. 2815–2824.","ama":"Kuzborskij I, Lampert C. Data-dependent stability of stochastic gradient descent. In: <i>Proceedings of the 35 Th International Conference on Machine Learning</i>. Vol 80. ML Research Press; 2018:2815-2824.","ista":"Kuzborskij I, Lampert C. 2018. Data-dependent stability of stochastic gradient descent. Proceedings of the 35 th International Conference on Machine Learning. ICML: International Conference on Machine Learning vol. 80, 2815–2824."},"arxiv":1,"publication":"Proceedings of the 35 th International Conference on Machine Learning","publisher":"ML Research Press","volume":80,"abstract":[{"lang":"eng","text":"We establish a data-dependent notion of algorithmic stability for Stochastic Gradient Descent (SGD), and employ it to develop novel generalization bounds. This is in contrast to previous distribution-free algorithmic stability results for SGD which depend on the worst-case constants. By virtue of the data-dependent argument, our bounds provide new insights into learning with SGD on convex and non-convex problems. In the convex case, we show that the bound on the generalization error depends on the risk at the initialization point. In the non-convex case, we prove that the expected curvature of the objective function around the initialization point has crucial influence on the generalization error. In both cases, our results suggest a simple data-driven strategy to stabilize SGD by pre-screening its initialization. As a corollary, our results allow us to show optimistic generalization bounds that exhibit fast convergence rates for SGD subject to a vanishing empirical risk and low noise of stochastic gradient. "}],"conference":{"start_date":"2018-07-10","name":"ICML: International Conference on Machine Learning","end_date":"2018-07-15","location":"Stockholm, Sweden"}}