---
_id: '18062'
abstract:
- lang: eng
  text: 'We explore the impact of parameter sparsity on the scaling behavior of Transformers
    trained on massive datasets (i.e., "foundation models"), in both vision and language
    domains. In this setting, we identify the first scaling law describing the relationship
    between weight sparsity, number of non-zero parameters, and amount of training
    data, which we validate empirically across model and data scales; on ViT/JFT-4B
    and T5/C4. These results allow us to characterize the "optimal sparsity", the
    sparsity level which yields the best performance for a given effective model size
    and training budget. For a fixed number of non-zero parameters, we identify that
    the optimal sparsity increases with the amount of data used for training. We also
    extend our study to different sparsity structures (such as the hardware-friendly
    n:m pattern) and strategies (such as starting from a pretrained dense model).
    Our findings shed light on the power and limitations of weight sparsity across
    various parameter and computational settings, offering both theoretical understanding
    and practical implications for leveraging sparsity towards computational efficiency
    improvements. We provide pruning and scaling law fitting code at: github.com/google-research/jaxpruner/tree/main/jaxpruner/projects/bigsparse.'
article_processing_charge: No
arxiv: 1
author:
- first_name: Elias
  full_name: Frantar, Elias
  id: 09a8f98d-ec99-11ea-ae11-c063a7b7fe5f
  last_name: Frantar
- first_name: Carlos Riquelme
  full_name: Ruiz, Carlos Riquelme
  last_name: Ruiz
- first_name: Neil
  full_name: Houlsby, Neil
  last_name: Houlsby
- first_name: Dan-Adrian
  full_name: Alistarh, Dan-Adrian
  id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
  last_name: Alistarh
  orcid: 0000-0003-3650-940X
- first_name: Utku
  full_name: Evci, Utku
  last_name: Evci
citation:
  ama: 'Frantar E, Ruiz CR, Houlsby N, Alistarh D-A, Evci U. Scaling laws for sparsely-connected
    foundation models. In: <i>The Twelfth International Conference on Learning Representations</i>.
    ; 2024.'
  apa: Frantar, E., Ruiz, C. R., Houlsby, N., Alistarh, D.-A., &#38; Evci, U. (2024).
    Scaling laws for sparsely-connected foundation models. In <i>The Twelfth International
    Conference on Learning Representations</i>. Vienna, Austria.
  chicago: Frantar, Elias, Carlos Riquelme Ruiz, Neil Houlsby, Dan-Adrian Alistarh,
    and Utku Evci. “Scaling Laws for Sparsely-Connected Foundation Models.” In <i>The
    Twelfth International Conference on Learning Representations</i>, 2024.
  ieee: E. Frantar, C. R. Ruiz, N. Houlsby, D.-A. Alistarh, and U. Evci, “Scaling
    laws for sparsely-connected foundation models,” in <i>The Twelfth International
    Conference on Learning Representations</i>, Vienna, Austria, 2024.
  ista: 'Frantar E, Ruiz CR, Houlsby N, Alistarh D-A, Evci U. 2024. Scaling laws for
    sparsely-connected foundation models. The Twelfth International Conference on
    Learning Representations. ICLR: International Conference on Learning Representations.'
  mla: Frantar, Elias, et al. “Scaling Laws for Sparsely-Connected Foundation Models.”
    <i>The Twelfth International Conference on Learning Representations</i>, 2024.
  short: E. Frantar, C.R. Ruiz, N. Houlsby, D.-A. Alistarh, U. Evci, in:, The Twelfth
    International Conference on Learning Representations, 2024.
conference:
  end_date: 2024-05-07
  location: Vienna, Austria
  name: 'ICLR: International Conference on Learning Representations'
  start_date: 2024-05-07
corr_author: '1'
date_created: 2024-09-13T10:31:08Z
date_published: 2024-01-16T00:00:00Z
date_updated: 2026-07-29T13:48:40Z
day: '16'
ddc:
- '000'
department:
- _id: DaAl
external_id:
  arxiv:
  - '2309.08520'
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://openreview.net/forum?id=i9K2ZWkYIP
month: '01'
oa: 1
oa_version: Published Version
publication: The Twelfth International Conference on Learning Representations
publication_status: published
quality_controlled: '1'
related_material:
  record:
  - id: '17485'
    relation: dissertation_contains
    status: public
scopus_import: '1'
status: public
title: Scaling laws for sparsely-connected foundation models
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2024'
...