---
res:
  bibo_abstract:
  - 'Communication-reduction techniques are a popular way to improve scalability in
    data-parallel training of deep neural networks (DNNs). The recent emergence of
    large language models such as GPT has created the need for new approaches to exploit
    data-parallelism. Among these, fully-sharded data parallel (FSDP) training is
    highly popular, yet it still encounters scalability bottlenecks. One reason is
    that applying compression techniques to FSDP is challenging: as the vast majority
    of the communication involves the model’s weights, direct compression alters convergence
    and leads to accuracy loss. We present QSDP, a variant of FSDP which supports
    both gradient and weight quantization with theoretical guarantees, is simple to
    implement and has essentially no overheads. To derive QSDP we prove that a natural
    modification of SGD achieves convergence even when we only maintain quantized
    weights, and thus the domain over which we train consists of quantized points
    and is, therefore, highly non-convex. We validate this approach by training GPT-family
    models with up to 1.3 billion parameters on a multi-node cluster. Experiments
    show that QSDP preserves model accuracy, while completely removing the communication
    bottlenecks of FSDP, providing end-to-end speedups of up to 2.2x.@eng'
  bibo_authorlist:
  - foaf_Person:
      foaf_givenName: Ilia
      foaf_name: Markov, Ilia
      foaf_surname: Markov
      foaf_workInfoHomepage: http://www.librecat.org/personId=D0CF4148-C985-11E9-8066-0BDEE5697425
  - foaf_Person:
      foaf_givenName: Adrian
      foaf_name: Vladu, Adrian
      foaf_surname: Vladu
  - foaf_Person:
      foaf_givenName: Qi
      foaf_name: Guo, Qi
      foaf_surname: Guo
  - foaf_Person:
      foaf_givenName: Dan-Adrian
      foaf_name: Alistarh, Dan-Adrian
      foaf_surname: Alistarh
      foaf_workInfoHomepage: http://www.librecat.org/personId=4A899BFC-F248-11E8-B48F-1D18A9856A87
    orcid: 0000-0003-3650-940X
  bibo_volume: 202
  dct_date: 2023^xs_gYear
  dct_isPartOf:
  - http://id.crossref.org/issn/2640-3498
  dct_language: eng
  dct_publisher: ML Research Press@
  dct_title: Quantized distributed training of large models with convergence guarantees@
...