{"author":[{"last_name":"Frantar","first_name":"Elias","id":"09a8f98d-ec99-11ea-ae11-c063a7b7fe5f","full_name":"Frantar, Elias"},{"last_name":"Ruiz","full_name":"Ruiz, Carlos Riquelme","first_name":"Carlos Riquelme"},{"last_name":"Houlsby","full_name":"Houlsby, Neil","first_name":"Neil"},{"orcid":"0000-0003-3650-940X","last_name":"Alistarh","id":"4A899BFC-F248-11E8-B48F-1D18A9856A87","first_name":"Dan-Adrian","full_name":"Alistarh, Dan-Adrian"},{"first_name":"Utku","full_name":"Evci, Utku","last_name":"Evci"}],"language":[{"iso":"eng"}],"year":"2024","quality_controlled":"1","date_published":"2024-01-16T00:00:00Z","date_created":"2024-09-13T10:31:08Z","date_updated":"2024-09-13T10:55:13Z","oa":1,"month":"01","publication":"The Twelfth International Conference on Learning Representations","title":"Scaling laws for sparsely-connected foundation models","citation":{"mla":"Frantar, Elias, et al. “Scaling Laws for Sparsely-Connected Foundation Models.” <i>The Twelfth International Conference on Learning Representations</i>, 2024.","ieee":"E. Frantar, C. R. Ruiz, N. Houlsby, D.-A. Alistarh, and U. Evci, “Scaling laws for sparsely-connected foundation models,” in <i>The Twelfth International Conference on Learning Representations</i>, Vienna, Austria, 2024.","ama":"Frantar E, Ruiz CR, Houlsby N, Alistarh D-A, Evci U. Scaling laws for sparsely-connected foundation models. In: <i>The Twelfth International Conference on Learning Representations</i>. ; 2024.","ista":"Frantar E, Ruiz CR, Houlsby N, Alistarh D-A, Evci U. 2024. Scaling laws for sparsely-connected foundation models. The Twelfth International Conference on Learning Representations. ICLR: International Conference on Learning Representations.","chicago":"Frantar, Elias, Carlos Riquelme Ruiz, Neil Houlsby, Dan-Adrian Alistarh, and Utku Evci. “Scaling Laws for Sparsely-Connected Foundation Models.” In <i>The Twelfth International Conference on Learning Representations</i>, 2024.","short":"E. Frantar, C.R. Ruiz, N. Houlsby, D.-A. Alistarh, U. Evci, in:, The Twelfth International Conference on Learning Representations, 2024.","apa":"Frantar, E., Ruiz, C. R., Houlsby, N., Alistarh, D.-A., &#38; Evci, U. (2024). Scaling laws for sparsely-connected foundation models. In <i>The Twelfth International Conference on Learning Representations</i>. Vienna, Austria."},"type":"conference","_id":"18062","conference":{"name":"ICLR: International Conference on Learning Representations","end_date":"2024-05-07","start_date":"2024-05-07","location":"Vienna, Austria"},"article_processing_charge":"No","oa_version":"Published Version","publication_status":"published","external_id":{"arxiv":["2309.08520"]},"day":"16","main_file_link":[{"url":"https://openreview.net/forum?id=i9K2ZWkYIP","open_access":"1"}],"abstract":[{"lang":"eng","text":"We explore the impact of parameter sparsity on the scaling behavior of Transformers trained on massive datasets (i.e., \"foundation models\"), in both vision and language domains. In this setting, we identify the first scaling law describing the relationship between weight sparsity, number of non-zero parameters, and amount of training data, which we validate empirically across model and data scales; on ViT/JFT-4B and T5/C4. These results allow us to characterize the \"optimal sparsity\", the sparsity level which yields the best performance for a given effective model size and training budget. For a fixed number of non-zero parameters, we identify that the optimal sparsity increases with the amount of data used for training. We also extend our study to different sparsity structures (such as the hardware-friendly n:m pattern) and strategies (such as starting from a pretrained dense model). Our findings shed light on the power and limitations of weight sparsity across various parameter and computational settings, offering both theoretical understanding and practical implications for leveraging sparsity towards computational efficiency improvements. We provide pruning and scaling law fitting code at: github.com/google-research/jaxpruner/tree/main/jaxpruner/projects/bigsparse."}],"status":"public","related_material":{"record":[{"status":"for_moderation","relation":"dissertation_contains","id":"17485"}]},"user_id":"8b945eb4-e2f2-11eb-945a-df72226e66a9","corr_author":"1","scopus_import":"1","department":[{"_id":"DaAl"}]}