---
OA_place: repository
OA_type: green
_id: '18976'
abstract:
- lang: eng
  text: We analyze asynchronous-type algorithms for distributed SGD in the heterogeneous
    setting, where each worker has its own computation and communication speeds, as
    well as data distribution. In these algorithms, workers compute possibly stale
    and stochastic gradients associated with their local data at some iteration back
    in history and then return those gradients to the server without synchronizing
    with other workers. We present a unified convergence theory for non-convex smooth
    functions in the heterogeneous regime. The proposed analysis provides convergence
    for pure asynchronous SGD and its various modifications. Moreover, our theory
    explains what affects the convergence rate and what can be done to improve the
    performance of asynchronous algorithms. In particular, we introduce a novel asynchronous
    method based on worker shuffling. As a by-product of our analysis, we also demonstrate
    convergence guarantees for gradient-type algorithms such as SGD with random reshuffling
    and shuffle-once mini-batch SGD. The derived rates match the best-known results
    for those algorithms, highlighting the tightness of our approach. Finally, our
    numerical evaluations support theoretical findings and show the good practical
    performance of our method.
acknowledgement: "The authors thank all anonymous reviewers for their valuable comments
  and suggestions on how to improve the manuscript. This work was done when Rustem
  Islamov was a Master’s student at Institut Polytechnique de Paris (IP Paris) and
  an intern at Institute of Science and Technology Austria (ISTA). The research of
  Rustem Islamov was supported by ISTA internship\r\nprogram. Mher Safaryan has received
  funding from the European Union’s Horizon 2020 research and innovation program under
  the Marie Skłodowska-Curie grant agreement No 101034413."
alternative_title:
- PMLR
article_processing_charge: No
arxiv: 1
author:
- first_name: Rustem
  full_name: Islamov, Rustem
  last_name: Islamov
- first_name: Mher
  full_name: Safaryan, Mher
  id: dd546b39-0804-11ed-9c55-ef075c39778d
  last_name: Safaryan
- first_name: Dan-Adrian
  full_name: Alistarh, Dan-Adrian
  id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
  last_name: Alistarh
  orcid: 0000-0003-3650-940X
citation:
  ama: 'Islamov R, Safaryan M, Alistarh D-A. AsGrad: A sharp unified analysis of asynchronous-SGD
    algorithms. In: <i>Proceedings of The 27th International Conference on Artificial
    Intelligence and Statistics</i>. Vol 238. ML Research Press; 2024:649-657.'
  apa: 'Islamov, R., Safaryan, M., &#38; Alistarh, D.-A. (2024). AsGrad: A sharp unified
    analysis of asynchronous-SGD algorithms. In <i>Proceedings of The 27th International
    Conference on Artificial Intelligence and Statistics</i> (Vol. 238, pp. 649–657).
    Valencia, Spain: ML Research Press.'
  chicago: 'Islamov, Rustem, Mher Safaryan, and Dan-Adrian Alistarh. “AsGrad: A Sharp
    Unified Analysis of Asynchronous-SGD Algorithms.” In <i>Proceedings of The 27th
    International Conference on Artificial Intelligence and Statistics</i>, 238:649–57.
    ML Research Press, 2024.'
  ieee: 'R. Islamov, M. Safaryan, and D.-A. Alistarh, “AsGrad: A sharp unified analysis
    of asynchronous-SGD algorithms,” in <i>Proceedings of The 27th International Conference
    on Artificial Intelligence and Statistics</i>, Valencia, Spain, 2024, vol. 238,
    pp. 649–657.'
  ista: 'Islamov R, Safaryan M, Alistarh D-A. 2024. AsGrad: A sharp unified analysis
    of asynchronous-SGD algorithms. Proceedings of The 27th International Conference
    on Artificial Intelligence and Statistics. AISTATS: Conference on Artificial Intelligence
    and Statistics, PMLR, vol. 238, 649–657.'
  mla: 'Islamov, Rustem, et al. “AsGrad: A Sharp Unified Analysis of Asynchronous-SGD
    Algorithms.” <i>Proceedings of The 27th International Conference on Artificial
    Intelligence and Statistics</i>, vol. 238, ML Research Press, 2024, pp. 649–57.'
  short: R. Islamov, M. Safaryan, D.-A. Alistarh, in:, Proceedings of The 27th International
    Conference on Artificial Intelligence and Statistics, ML Research Press, 2024,
    pp. 649–657.
conference:
  end_date: 2024-05-04
  location: Valencia, Spain
  name: 'AISTATS: Conference on Artificial Intelligence and Statistics'
  start_date: 2024-05-02
corr_author: '1'
date_created: 2025-01-30T08:15:49Z
date_published: 2024-05-15T00:00:00Z
date_updated: 2025-04-14T07:54:52Z
day: '15'
department:
- _id: DaAl
ec_funded: 1
external_id:
  arxiv:
  - '2310.20452'
intvolume: '       238'
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://doi.org/10.48550/arXiv.2310.20452
month: '05'
oa: 1
oa_version: Preprint
page: 649-657
project:
- _id: fc2ed2f7-9c52-11eb-aca3-c01059dda49c
  call_identifier: H2020
  grant_number: '101034413'
  name: 'IST-BRIDGE: International postdoctoral program'
publication: Proceedings of The 27th International Conference on Artificial Intelligence
  and Statistics
publication_identifier:
  eissn:
  - 2640-3498
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'AsGrad: A sharp unified analysis of asynchronous-SGD algorithms'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 238
year: '2024'
...