---
_id: '11436'
abstract:
- lang: eng
  text: Asynchronous distributed algorithms are a popular way to reduce synchronization
    costs in large-scale optimization, and in particular for neural network training.
    However, for nonsmooth and nonconvex objectives, few convergence guarantees exist
    beyond cases where closed-form proximal operator solutions are available. As training
    most popular deep neural networks corresponds to optimizing nonsmooth and nonconvex
    objectives, there is a pressing need for such convergence guarantees. In this
    paper, we analyze for the first time the convergence of stochastic asynchronous
    optimization for this general class of objectives. In particular, we focus on
    stochastic subgradient methods allowing for block variable partitioning, where
    the shared model is asynchronously updated by concurrent processes. To this end,
    we use a probabilistic model which captures key features of real asynchronous
    scheduling between concurrent processes. Under this model, we establish convergence
    with probability one to an invariant set for stochastic subgradient methods with
    momentum. From a practical perspective, one issue with the family of algorithms
    that we consider is that they are not efficiently supported by machine learning
    frameworks, which mostly focus on distributed data-parallel strategies. To address
    this, we propose a new implementation strategy for shared-memory based training
    of deep neural networks for a partitioned but shared model in single- and multi-GPU
    settings. Based on this implementation, we achieve on average1.2x speed-up in
    comparison to state-of-the-art training methods for popular image classification
    tasks, without compromising accuracy.
acknowledgement: Vyacheslav Kungurtsev was supported by the OP VVV project CZ.02.1.01/0.0/0.0/16
  019/0000765 “Research Center for Informatics. Bapi Chatterjee was supported by the
  European Union’s Horizon 2020 research and innovation programme under the Marie
  Sklodowska-Curie grant agreement No. 754411 (ISTPlus). Dan Alistarh has received
  funding from the European Research Council (ERC) under the European Union’s Horizon
  2020 research and innovation programme (grant agreement No 805223 ScaleML).
article_processing_charge: No
arxiv: 1
author:
- first_name: Vyacheslav
  full_name: Kungurtsev, Vyacheslav
  last_name: Kungurtsev
- first_name: Malcolm
  full_name: Egan, Malcolm
  last_name: Egan
- first_name: Bapi
  full_name: Chatterjee, Bapi
  id: 3C41A08A-F248-11E8-B48F-1D18A9856A87
  last_name: Chatterjee
  orcid: 0000-0002-2742-4028
- first_name: Dan-Adrian
  full_name: Alistarh, Dan-Adrian
  id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
  last_name: Alistarh
  orcid: 0000-0003-3650-940X
citation:
  ama: 'Kungurtsev V, Egan M, Chatterjee B, Alistarh D-A. Asynchronous optimization
    methods for efficient training of deep neural networks with guarantees. In: <i>35th
    AAAI Conference on Artificial Intelligence, AAAI 2021</i>. Vol 35. AAAI Press;
    2021:8209-8216.'
  apa: 'Kungurtsev, V., Egan, M., Chatterjee, B., &#38; Alistarh, D.-A. (2021). Asynchronous
    optimization methods for efficient training of deep neural networks with guarantees.
    In <i>35th AAAI Conference on Artificial Intelligence, AAAI 2021</i> (Vol. 35,
    pp. 8209–8216). Virtual, Online: AAAI Press.'
  chicago: Kungurtsev, Vyacheslav, Malcolm Egan, Bapi Chatterjee, and Dan-Adrian Alistarh.
    “Asynchronous Optimization Methods for Efficient Training of Deep Neural Networks
    with Guarantees.” In <i>35th AAAI Conference on Artificial Intelligence, AAAI
    2021</i>, 35:8209–16. AAAI Press, 2021.
  ieee: V. Kungurtsev, M. Egan, B. Chatterjee, and D.-A. Alistarh, “Asynchronous optimization
    methods for efficient training of deep neural networks with guarantees,” in <i>35th
    AAAI Conference on Artificial Intelligence, AAAI 2021</i>, Virtual, Online, 2021,
    vol. 35, no. 9B, pp. 8209–8216.
  ista: 'Kungurtsev V, Egan M, Chatterjee B, Alistarh D-A. 2021. Asynchronous optimization
    methods for efficient training of deep neural networks with guarantees. 35th AAAI
    Conference on Artificial Intelligence, AAAI 2021. AAAI: Conference on Artificial
    Intelligence vol. 35, 8209–8216.'
  mla: Kungurtsev, Vyacheslav, et al. “Asynchronous Optimization Methods for Efficient
    Training of Deep Neural Networks with Guarantees.” <i>35th AAAI Conference on
    Artificial Intelligence, AAAI 2021</i>, vol. 35, no. 9B, AAAI Press, 2021, pp.
    8209–16.
  short: V. Kungurtsev, M. Egan, B. Chatterjee, D.-A. Alistarh, in:, 35th AAAI Conference
    on Artificial Intelligence, AAAI 2021, AAAI Press, 2021, pp. 8209–8216.
conference:
  end_date: 2021-02-09
  location: Virtual, Online
  name: 'AAAI: Conference on Artificial Intelligence'
  start_date: 2021-02-02
date_created: 2022-06-05T22:01:52Z
date_published: 2021-05-18T00:00:00Z
date_updated: 2025-04-14T07:43:57Z
day: '18'
department:
- _id: DaAl
ec_funded: 1
external_id:
  arxiv:
  - '1905.11845'
intvolume: '        35'
issue: 9B
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: ' https://doi.org/10.48550/arXiv.1905.11845'
month: '05'
oa: 1
oa_version: Preprint
page: 8209-8216
project:
- _id: 260C2330-B435-11E9-9278-68D0E5697425
  call_identifier: H2020
  grant_number: '754411'
  name: ISTplus - Postdoctoral Fellowships
- _id: 268A44D6-B435-11E9-9278-68D0E5697425
  call_identifier: H2020
  grant_number: '805223'
  name: Elastic Coordination for Scalable Machine Learning
publication: 35th AAAI Conference on Artificial Intelligence, AAAI 2021
publication_identifier:
  eissn:
  - 2374-3468
  isbn:
  - '9781713835974'
  issn:
  - 2159-5399
publication_status: published
publisher: AAAI Press
quality_controlled: '1'
scopus_import: '1'
status: public
title: Asynchronous optimization methods for efficient training of deep neural networks
  with guarantees
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2021'
...