---
res:
  bibo_abstract:
  - The growing energy and performance costs of deep learning have driven the community
    to reduce the size of neural networks by selectively pruning components. Similarly
    to their biological counterparts, sparse networks generalize just as well, sometimes
    even better than, the original dense networks. Sparsity promises to reduce the
    memory footprint of regular networks to fit mobile devices, as well as shorten
    training time for ever growing networks. In this paper, we survey prior work on
    sparsity in deep learning and provide an extensive tutorial of sparsification
    for both inference and training. We describe approaches to remove and add elements
    of neural networks, different training strategies to achieve model sparsity, and
    mechanisms to exploit sparsity in practice. Our work distills ideas from more
    than 300 research papers and provides guidance to practitioners who wish to utilize
    sparsity today, as well as to researchers whose goal is to push the frontier forward.
    We include the necessary background on mathematical methods in sparsification,
    describe phenomena such as early structure adaptation, the intricate relations
    between sparsity and the training process, and show techniques for achieving acceleration
    on real hardware. We also define a metric of pruned parameter efficiency that
    could serve as a baseline for comparison of different sparse networks. We close
    by speculating on how sparsity can improve future workloads and outline major
    open problems in the field.@eng
  bibo_authorlist:
  - foaf_Person:
      foaf_givenName: Torsten
      foaf_name: Hoefler, Torsten
      foaf_surname: Hoefler
  - foaf_Person:
      foaf_givenName: Dan-Adrian
      foaf_name: Alistarh, Dan-Adrian
      foaf_surname: Alistarh
      foaf_workInfoHomepage: http://www.librecat.org/personId=4A899BFC-F248-11E8-B48F-1D18A9856A87
    orcid: 0000-0003-3650-940X
  - foaf_Person:
      foaf_givenName: Tal
      foaf_name: Ben-Nun, Tal
      foaf_surname: Ben-Nun
  - foaf_Person:
      foaf_givenName: Nikoli
      foaf_name: Dryden, Nikoli
      foaf_surname: Dryden
  - foaf_Person:
      foaf_givenName: Elena-Alexandra
      foaf_name: Peste, Elena-Alexandra
      foaf_surname: Peste
      foaf_workInfoHomepage: http://www.librecat.org/personId=32D78294-F248-11E8-B48F-1D18A9856A87
  bibo_issue: '241'
  bibo_volume: 22
  dct_date: 2021^xs_gYear
  dct_isPartOf:
  - http://id.crossref.org/issn/1532-4435
  - http://id.crossref.org/issn/1533-7928
  dct_language: eng
  dct_publisher: ML Research Press@
  dct_title: 'Sparsity in deep learning: Pruning and growth for efficient inference
    and training in neural networks@'
...