{"article_processing_charge":"No","author":[{"last_name":"Islamov","full_name":"Islamov, Rustem","first_name":"Rustem"},{"full_name":"Safaryan, Mher","first_name":"Mher","last_name":"Safaryan","id":"dd546b39-0804-11ed-9c55-ef075c39778d"},{"full_name":"Alistarh, Dan-Adrian","first_name":"Dan-Adrian","orcid":"0000-0003-3650-940X","id":"4A899BFC-F248-11E8-B48F-1D18A9856A87","last_name":"Alistarh"}],"title":"AsGrad: A sharp unified analysis of asynchronous-SGD algorithms","ec_funded":1,"alternative_title":["PMLR"],"abstract":[{"text":"We analyze asynchronous-type algorithms for distributed SGD in the heterogeneous setting, where each worker has its own computation and communication speeds, as well as data distribution. In these algorithms, workers compute possibly stale and stochastic gradients associated with their local data at some iteration back in history and then return those gradients to the server without synchronizing with other workers. We present a unified convergence theory for non-convex smooth functions in the heterogeneous regime. The proposed analysis provides convergence for pure asynchronous SGD and its various modifications. Moreover, our theory explains what affects the convergence rate and what can be done to improve the performance of asynchronous algorithms. In particular, we introduce a novel asynchronous method based on worker shuffling. As a by-product of our analysis, we also demonstrate convergence guarantees for gradient-type algorithms such as SGD with random reshuffling and shuffle-once mini-batch SGD. The derived rates match the best-known results for those algorithms, highlighting the tightness of our approach. Finally, our numerical evaluations support theoretical findings and show the good practical performance of our method.","lang":"eng"}],"department":[{"_id":"DaAl"}],"citation":{"ista":"Islamov R, Safaryan M, Alistarh D-A. 2024. AsGrad: A sharp unified analysis of asynchronous-SGD algorithms. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics. AISTATS: Conference on Artificial Intelligence and Statistics, PMLR, vol. 238, 649–657.","mla":"Islamov, Rustem, et al. “AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithms.” <i>Proceedings of The 27th International Conference on Artificial Intelligence and Statistics</i>, vol. 238, ML Research Press, 2024, pp. 649–57.","apa":"Islamov, R., Safaryan, M., &#38; Alistarh, D.-A. (2024). AsGrad: A sharp unified analysis of asynchronous-SGD algorithms. In <i>Proceedings of The 27th International Conference on Artificial Intelligence and Statistics</i> (Vol. 238, pp. 649–657). Valencia, Spain: ML Research Press.","ieee":"R. Islamov, M. Safaryan, and D.-A. Alistarh, “AsGrad: A sharp unified analysis of asynchronous-SGD algorithms,” in <i>Proceedings of The 27th International Conference on Artificial Intelligence and Statistics</i>, Valencia, Spain, 2024, vol. 238, pp. 649–657.","chicago":"Islamov, Rustem, Mher Safaryan, and Dan-Adrian Alistarh. “AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithms.” In <i>Proceedings of The 27th International Conference on Artificial Intelligence and Statistics</i>, 238:649–57. ML Research Press, 2024.","ama":"Islamov R, Safaryan M, Alistarh D-A. AsGrad: A sharp unified analysis of asynchronous-SGD algorithms. In: <i>Proceedings of The 27th International Conference on Artificial Intelligence and Statistics</i>. Vol 238. ML Research Press; 2024:649-657.","short":"R. Islamov, M. Safaryan, D.-A. Alistarh, in:, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, ML Research Press, 2024, pp. 649–657."},"conference":{"location":"Valencia, Spain","name":"AISTATS: Conference on Artificial Intelligence and Statistics","start_date":"2024-05-02","end_date":"2024-05-04"},"publication_identifier":{"eissn":["2640-3498"]},"user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","day":"15","type":"conference","arxiv":1,"scopus_import":"1","year":"2024","oa_version":"Preprint","date_created":"2025-01-30T08:15:49Z","project":[{"name":"IST-BRIDGE: International postdoctoral program","call_identifier":"H2020","_id":"fc2ed2f7-9c52-11eb-aca3-c01059dda49c","grant_number":"101034413"}],"oa":1,"date_updated":"2025-01-30T08:16:31Z","main_file_link":[{"url":"https://doi.org/10.48550/arXiv.2310.20452","open_access":"1"}],"date_published":"2024-05-15T00:00:00Z","language":[{"iso":"eng"}],"_id":"18976","publication_status":"published","quality_controlled":"1","page":"649-657","publication":"Proceedings of The 27th International Conference on Artificial Intelligence and Statistics","volume":238,"corr_author":"1","intvolume":"       238","external_id":{"arxiv":["2310.20452"]},"OA_type":"green","acknowledgement":"The authors thank all anonymous reviewers for their valuable comments and suggestions on how to improve the manuscript. This work was done when Rustem Islamov was a Master’s student at Institut Polytechnique de Paris (IP Paris) and an intern at Institute of Science and Technology Austria (ISTA). The research of Rustem Islamov was supported by ISTA internship\r\nprogram. Mher Safaryan has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 101034413.","publisher":"ML Research Press","OA_place":"repository","month":"05","status":"public"}