---
res:
  bibo_abstract:
  - Parallel SGD in a shared-memory setting is oft-represented by the popular Hogwild!
    algorithm, in which lock-free updates are asynchronously performed by multiple
    computing processes. Unfortunately, scaling Hogwild! to distributed workers is
    largely unexplored. Specifically, it is unknown if any adaptation of Hogwild!
    to the popular decentralized multi-GPU setting offers any competitive speedup,
    either empirically or theoretically. In this work, we investigate the potential
    of decentralizing Hogwild! by incorporating simultaneously (a) asynchronous local
    gradient updates on the shared memory of GPUs, and (b) non-blocking asynchronous
    decentralized federated averaging. A naive direct implementation shows degradation
    in performance, arising from scheduling overheads and concurrent write conflicts
    on GPUs. To mitigate these drawbacks, we investigate and propose a new method,
    based on careful block selection rules, which update only portions of the parameter
    vectors. Our experiments show that the resulting decentralized training method
    exhibits improved throughput and competitive accuracy for standard image classification
    benchmarks on the CIFAR-10, CIFAR-100, and Imagenet datasets. On the theoretical
    side, we prove that our method guarantees sublinear ergodic convergence rates
    for non-convex objectives.@eng
  bibo_authorlist:
  - foaf_Person:
      foaf_givenName: Bapi
      foaf_name: Chatterjee, Bapi
      foaf_surname: Chatterjee
      foaf_workInfoHomepage: http://www.librecat.org/personId=3C41A08A-F248-11E8-B48F-1D18A9856A87
    orcid: 0000-0002-2742-4028
  - foaf_Person:
      foaf_givenName: Vyacheslav
      foaf_name: Kungurtsev, Vyacheslav
      foaf_surname: Kungurtsev
  - foaf_Person:
      foaf_givenName: Dan-Adrian
      foaf_name: Alistarh, Dan-Adrian
      foaf_surname: Alistarh
      foaf_workInfoHomepage: http://www.librecat.org/personId=4A899BFC-F248-11E8-B48F-1D18A9856A87
    orcid: 0000-0003-3650-940X
  bibo_doi: 10.1109/ICDCS60910.2024.00084
  dct_date: 2024^xs_gYear
  dct_identifier:
  - UT:001304430200075
  dct_isPartOf:
  - http://id.crossref.org/issn/1063-6927
  - http://id.crossref.org/issn/2575-8411
  - http://id.crossref.org/issn/9798350386059
  dct_language: eng
  dct_publisher: IEEE@
  dct_title: Federated SGD with local asynchrony@
...