[{"file_date_updated":"2025-08-04T08:39:51Z","language":[{"iso":"eng"}],"month":"04","day":"01","publication":"13th International Conference on Learning Representations","oa_version":"Published Version","article_processing_charge":"No","abstract":[{"text":"We introduce LDAdam, a memory-efficient optimizer for training large models, that performs adaptive optimization steps within lower dimensional subspaces, while consistently exploring the full parameter space during training. This strategy keeps the optimizer's memory footprint to a fraction of the model size. LDAdam relies on a new projection-aware update rule for the optimizer states that allows for transitioning between subspaces, i.e., estimation of the statistics of the projected gradients. To mitigate the errors due to low-rank projection, LDAdam integrates a new generalized error feedback mechanism, which explicitly accounts for both gradient and optimizer state compression. We prove the convergence of LDAdam under standard assumptions, and provide empirical evidence that LDAdam allows for efficient fine-tuning and pre-training of language models.","lang":"eng"}],"_id":"20034","type":"conference","OA_type":"diamond","citation":{"ieee":"T. Robert, M. Safaryan, I.-V. Modoranu, and D.-A. Alistarh, “LDAdam: Adaptive optimization from low-dimensional gradient statistics,” in <i>13th International Conference on Learning Representations</i>, Singapore, Singapore, 2025, pp. 101877–101913.","ista":"Robert T, Safaryan M, Modoranu I-V, Alistarh D-A. 2025. LDAdam: Adaptive optimization from low-dimensional gradient statistics. 13th International Conference on Learning Representations. ICLR: International Conference on Learning Representations, 101877–101913.","apa":"Robert, T., Safaryan, M., Modoranu, I.-V., &#38; Alistarh, D.-A. (2025). LDAdam: Adaptive optimization from low-dimensional gradient statistics. In <i>13th International Conference on Learning Representations</i> (pp. 101877–101913). Singapore, Singapore: ICLR.","short":"T. Robert, M. Safaryan, I.-V. Modoranu, D.-A. Alistarh, in:, 13th International Conference on Learning Representations, ICLR, 2025, pp. 101877–101913.","chicago":"Robert, Thomas, Mher Safaryan, Ionut-Vlad Modoranu, and Dan-Adrian Alistarh. “LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics.” In <i>13th International Conference on Learning Representations</i>, 101877–913. ICLR, 2025.","mla":"Robert, Thomas, et al. “LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics.” <i>13th International Conference on Learning Representations</i>, ICLR, 2025, pp. 101877–913.","ama":"Robert T, Safaryan M, Modoranu I-V, Alistarh D-A. LDAdam: Adaptive optimization from low-dimensional gradient statistics. In: <i>13th International Conference on Learning Representations</i>. ICLR; 2025:101877-101913."},"page":"101877-101913","oa":1,"user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","date_updated":"2025-08-04T08:41:10Z","scopus_import":"1","title":"LDAdam: Adaptive optimization from low-dimensional gradient statistics","date_published":"2025-04-01T00:00:00Z","related_material":{"link":[{"url":"https://github.com/IST-DASLab/LDAdam","relation":"software"}]},"publication_identifier":{"isbn":["9798331320850"]},"date_created":"2025-07-20T22:02:02Z","quality_controlled":"1","ddc":["000"],"publisher":"ICLR","author":[{"last_name":"Robert","full_name":"Robert, Thomas","first_name":"Thomas"},{"last_name":"Safaryan","full_name":"Safaryan, Mher","id":"dd546b39-0804-11ed-9c55-ef075c39778d","first_name":"Mher"},{"id":"449f7a18-f128-11eb-9611-9b430c0c6333","first_name":"Ionut-Vlad","last_name":"Modoranu","full_name":"Modoranu, Ionut-Vlad"},{"first_name":"Dan-Adrian","id":"4A899BFC-F248-11E8-B48F-1D18A9856A87","full_name":"Alistarh, Dan-Adrian","last_name":"Alistarh","orcid":"0000-0003-3650-940X"}],"OA_place":"publisher","year":"2025","status":"public","tmp":{"short":"CC BY (4.0)","image":"/images/cc_by.png","legal_code_url":"https://creativecommons.org/licenses/by/4.0/legalcode","name":"Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)"},"file":[{"access_level":"open_access","date_updated":"2025-08-04T08:39:51Z","file_id":"20113","date_created":"2025-08-04T08:39:51Z","content_type":"application/pdf","success":1,"file_name":"2025_ICLR_Robert.pdf","relation":"main_file","checksum":"9327d82569358d7bf1c3ec1a9952e721","creator":"dernst","file_size":1346111}],"arxiv":1,"external_id":{"arxiv":["2410.16103"]},"publication_status":"published","corr_author":"1","has_accepted_license":"1","department":[{"_id":"DaAl"}],"conference":{"start_date":"2025-04-24","name":"ICLR: International Conference on Learning Representations","end_date":"2025-04-28","location":"Singapore, Singapore"}},{"language":[{"iso":"eng"}],"month":"07","day":"30","volume":235,"publication":"41st International Conference on Machine Learning","type":"conference","OA_type":"green","article_processing_charge":"No","oa_version":"Preprint","_id":"18975","abstract":[{"text":"Leveraging second-order information about the loss at the scale of deep networks is one of the main lines of approach for improving the performance of current optimizers for deep learning. Yet, existing approaches for accurate full-matrix preconditioning, such as Full-Matrix Adagrad (GGT) or Matrix-Free Approximate Curvature (M-FAC) suffer from massive storage costs when applied even to small-scale models, as they must store a sliding window of gradients, whose memory requirements are multiplicative in the model dimension. In this paper, we address this issue via a novel and efficient error-feedback technique that can be applied to compress preconditioners by up to two orders of magnitude in practice, without loss of convergence. Specifically, our approach compresses the gradient information via sparsification or low-rank compression before it is fed into the preconditioner, feeding the compression error back into future iterations. Extensive experiments on deep neural networks show that this approach can compress full-matrix preconditioners to up to 99% sparsity without accuracy loss, effectively removing the memory overhead of fullmatrix preconditioners such as GGT and M-FAC.","lang":"eng"}],"citation":{"ieee":"I.-V. Modoranu, A. Kalinov, E. Kurtic, E. Frantar, and D.-A. Alistarh, “Error feedback can accurately compress preconditioners,” in <i>41st International Conference on Machine Learning</i>, Vienna, Austria, 2024, vol. 235, pp. 35910–35933.","apa":"Modoranu, I.-V., Kalinov, A., Kurtic, E., Frantar, E., &#38; Alistarh, D.-A. (2024). Error feedback can accurately compress preconditioners. In <i>41st International Conference on Machine Learning</i> (Vol. 235, pp. 35910–35933). Vienna, Austria: ML Research Press.","ista":"Modoranu I-V, Kalinov A, Kurtic E, Frantar E, Alistarh D-A. 2024. Error feedback can accurately compress preconditioners. 41st International Conference on Machine Learning. ICML: International Conference on Machine Learning, PMLR, vol. 235, 35910–35933.","short":"I.-V. Modoranu, A. Kalinov, E. Kurtic, E. Frantar, D.-A. Alistarh, in:, 41st International Conference on Machine Learning, ML Research Press, 2024, pp. 35910–35933.","chicago":"Modoranu, Ionut-Vlad, Aleksei Kalinov, Eldar Kurtic, Elias Frantar, and Dan-Adrian Alistarh. “Error Feedback Can Accurately Compress Preconditioners.” In <i>41st International Conference on Machine Learning</i>, 235:35910–33. ML Research Press, 2024.","mla":"Modoranu, Ionut-Vlad, et al. “Error Feedback Can Accurately Compress Preconditioners.” <i>41st International Conference on Machine Learning</i>, vol. 235, ML Research Press, 2024, pp. 35910–33.","ama":"Modoranu I-V, Kalinov A, Kurtic E, Frantar E, Alistarh D-A. Error feedback can accurately compress preconditioners. In: <i>41st International Conference on Machine Learning</i>. Vol 235. ML Research Press; 2024:35910-35933."},"page":"35910-35933","date_updated":"2025-01-30T07:54:16Z","acknowledgement":"The authors thank Adrian Vladu, Razvan Pascanu, Alexandra Peste, Mher Safaryan for their valuable feedback, the IT department from Institute of Science and Technology Austria for the hardware support and Weights and Biases for the infrastructure to track all our experiments.","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","oa":1,"title":"Error feedback can accurately compress preconditioners","main_file_link":[{"url":"https://doi.org/10.48550/arXiv.2306.06098","open_access":"1"}],"acknowledged_ssus":[{"_id":"CampIT"}],"scopus_import":"1","date_published":"2024-07-30T00:00:00Z","date_created":"2025-01-30T07:53:22Z","quality_controlled":"1","publication_identifier":{"eissn":["2640-3498"]},"year":"2024","intvolume":"       235","publisher":"ML Research Press","OA_place":"repository","author":[{"full_name":"Modoranu, Ionut-Vlad","last_name":"Modoranu","first_name":"Ionut-Vlad","id":"449f7a18-f128-11eb-9611-9b430c0c6333"},{"full_name":"Kalinov, Aleksei","orcid":"0000-0003-2189-3904","last_name":"Kalinov","id":"44b7120e-eb97-11eb-a6c2-e1557aa81d02","first_name":"Aleksei"},{"last_name":"Kurtic","full_name":"Kurtic, Eldar","first_name":"Eldar","id":"47beb3a5-07b5-11eb-9b87-b108ec578218"},{"last_name":"Frantar","full_name":"Frantar, Elias","id":"09a8f98d-ec99-11ea-ae11-c063a7b7fe5f","first_name":"Elias"},{"id":"4A899BFC-F248-11E8-B48F-1D18A9856A87","first_name":"Dan-Adrian","full_name":"Alistarh, Dan-Adrian","orcid":"0000-0003-3650-940X","last_name":"Alistarh"}],"alternative_title":["PMLR"],"status":"public","arxiv":1,"department":[{"_id":"DaAl"}],"corr_author":"1","external_id":{"arxiv":["2306.06098"]},"publication_status":"published","conference":{"location":"Vienna, Austria","end_date":"2024-07-27","name":"ICML: International Conference on Machine Learning","start_date":"2024-07-21"}},{"publisher":"Neural Information Processing Systems Foundation","OA_place":"repository","author":[{"last_name":"Modoranu","full_name":"Modoranu, Ionut-Vlad","first_name":"Ionut-Vlad","id":"449f7a18-f128-11eb-9611-9b430c0c6333"},{"last_name":"Safaryan","full_name":"Safaryan, Mher","first_name":"Mher","id":"dd546b39-0804-11ed-9c55-ef075c39778d"},{"full_name":"Malinovsky, Grigory","last_name":"Malinovsky","first_name":"Grigory"},{"full_name":"Kurtic, Eldar","last_name":"Kurtic","first_name":"Eldar","id":"47beb3a5-07b5-11eb-9b87-b108ec578218"},{"last_name":"Robert","full_name":"Robert, Thomas","first_name":"Thomas","id":"de632733-1457-11f0-ae22-b5914b8c1c41"},{"first_name":"Peter","full_name":"Richtárik, Peter","last_name":"Richtárik"},{"last_name":"Alistarh","full_name":"Alistarh, Dan-Adrian","orcid":"0000-0003-3650-940X","first_name":"Dan-Adrian","id":"4A899BFC-F248-11E8-B48F-1D18A9856A87"}],"year":"2024","intvolume":"        37","publication_identifier":{"issn":["1049-5258"]},"ec_funded":1,"date_created":"2025-04-06T22:01:32Z","quality_controlled":"1","date_published":"2024-12-20T00:00:00Z","related_material":{"link":[{"relation":"software","url":"https://github.com/IST-DASLab/MicroAdam"}]},"main_file_link":[{"open_access":"1","url":"https://doi.org/10.48550/arXiv.2405.15593"}],"acknowledged_ssus":[{"_id":"CampIT"}],"scopus_import":"1","title":"MICROADAM: Accurate adaptive optimization with low space overhead and provable convergence","publication_status":"published","corr_author":"1","external_id":{"arxiv":["2405.15593"]},"department":[{"_id":"DaAl"}],"arxiv":1,"status":"public","alternative_title":["Advances in Neural Information Processing Systems"],"day":"20","volume":37,"language":[{"iso":"eng"}],"month":"12","acknowledgement":"The authors thank Razvan Pascanu, Mahdi Nikdan and Soroush Tabesh for their valuable feedback, the IT department from Institute of Science and Technology Austria for the hardware support and Weights and Biases for the infrastructure to track all our experiments. Mher Safaryan has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement No 101034413.","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","oa":1,"date_updated":"2025-05-14T11:32:52Z","citation":{"mla":"Modoranu, Ionut-Vlad, et al. “MICROADAM: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence.” <i>38th Conference on Neural Information Processing Systems</i>, vol. 37, Neural Information Processing Systems Foundation, 2024.","chicago":"Modoranu, Ionut-Vlad, Mher Safaryan, Grigory Malinovsky, Eldar Kurtic, Thomas Robert, Peter Richtárik, and Dan-Adrian Alistarh. “MICROADAM: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence.” In <i>38th Conference on Neural Information Processing Systems</i>, Vol. 37. Neural Information Processing Systems Foundation, 2024.","ama":"Modoranu I-V, Safaryan M, Malinovsky G, et al. MICROADAM: Accurate adaptive optimization with low space overhead and provable convergence. In: <i>38th Conference on Neural Information Processing Systems</i>. Vol 37. Neural Information Processing Systems Foundation; 2024.","ieee":"I.-V. Modoranu <i>et al.</i>, “MICROADAM: Accurate adaptive optimization with low space overhead and provable convergence,” in <i>38th Conference on Neural Information Processing Systems</i>, 2024, vol. 37.","short":"I.-V. Modoranu, M. Safaryan, G. Malinovsky, E. Kurtic, T. Robert, P. Richtárik, D.-A. Alistarh, in:, 38th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2024.","ista":"Modoranu I-V, Safaryan M, Malinovsky G, Kurtic E, Robert T, Richtárik P, Alistarh D-A. 2024. MICROADAM: Accurate adaptive optimization with low space overhead and provable convergence. 38th Conference on Neural Information Processing Systems. , Advances in Neural Information Processing Systems, vol. 37.","apa":"Modoranu, I.-V., Safaryan, M., Malinovsky, G., Kurtic, E., Robert, T., Richtárik, P., &#38; Alistarh, D.-A. (2024). MICROADAM: Accurate adaptive optimization with low space overhead and provable convergence. In <i>38th Conference on Neural Information Processing Systems</i> (Vol. 37). Neural Information Processing Systems Foundation."},"oa_version":"Preprint","article_processing_charge":"No","abstract":[{"text":"We propose a new variant of the Adam optimizer [Kingma and Ba, 2014] called\r\nMICROADAM that specifically minimizes memory overheads, while maintaining\r\ntheoretical convergence guarantees. We achieve this by compressing the gradient\r\ninformation before it is fed into the optimizer state, thereby reducing its memory\r\nfootprint significantly. We control the resulting compression error via a novel\r\ninstance of the classical error feedback mechanism from distributed optimization [Seide et al., 2014, Alistarh et al., 2018, Karimireddy et al., 2019] in which\r\nthe error correction information is itself compressed to allow for practical memory\r\ngains. We prove that the resulting approach maintains theoretical convergence\r\nguarantees competitive to those of AMSGrad, while providing good practical performance. Specifically, we show that MICROADAM can be implemented efficiently\r\non GPUs: on both million-scale (BERT) and billion-scale (LLaMA) models, MICROADAM provides practical convergence competitive to that of the uncompressed\r\nAdam baseline, with lower memory usage and similar running time. Our code is\r\navailable at https://github.com/IST-DASLab/MicroAdam.","lang":"eng"}],"_id":"19510","project":[{"_id":"fc2ed2f7-9c52-11eb-aca3-c01059dda49c","name":"IST-BRIDGE: International postdoctoral program","grant_number":"101034413","call_identifier":"H2020"}],"type":"conference","OA_type":"green","publication":"38th Conference on Neural Information Processing Systems"},{"title":"The iterative optimal brain surgeon: Faster sparse recovery by leveraging second-order information","acknowledged_ssus":[{"_id":"CampIT"}],"scopus_import":"1","main_file_link":[{"open_access":"1","url":"https://doi.org/10.48550/arXiv.2408.17163"}],"date_published":"2024-12-20T00:00:00Z","quality_controlled":"1","date_created":"2025-04-06T22:01:32Z","ec_funded":1,"publication_identifier":{"issn":["1049-5258"]},"intvolume":"        37","year":"2024","author":[{"first_name":"Diyuan","id":"1a5914c2-896a-11ed-bdf8-fb80621a0635","last_name":"Wu","full_name":"Wu, Diyuan"},{"last_name":"Modoranu","full_name":"Modoranu, Ionut-Vlad","first_name":"Ionut-Vlad","id":"449f7a18-f128-11eb-9611-9b430c0c6333"},{"last_name":"Safaryan","full_name":"Safaryan, Mher","first_name":"Mher","id":"dd546b39-0804-11ed-9c55-ef075c39778d"},{"first_name":"Denis","full_name":"Kuznedelev, Denis","last_name":"Kuznedelev"},{"full_name":"Alistarh, Dan-Adrian","last_name":"Alistarh","orcid":"0000-0003-3650-940X","id":"4A899BFC-F248-11E8-B48F-1D18A9856A87","first_name":"Dan-Adrian"}],"OA_place":"repository","publisher":"Neural Information Processing Systems Foundation","alternative_title":["Advances in Neural Information Processing Systems"],"status":"public","arxiv":1,"department":[{"_id":"DaAl"},{"_id":"MaMo"}],"external_id":{"arxiv":["2408.17163"]},"corr_author":"1","publication_status":"published","conference":{"location":"Vancouver, Canada","end_date":"2024-12-15","name":"NeurIPS: Neural Information Processing Systems","start_date":"2024-12-09"},"month":"12","language":[{"iso":"eng"}],"volume":37,"day":"20","publication":"38th Conference on Neural Information Processing Systems","OA_type":"green","project":[{"_id":"fc2ed2f7-9c52-11eb-aca3-c01059dda49c","name":"IST-BRIDGE: International postdoctoral program","grant_number":"101034413","call_identifier":"H2020"}],"type":"conference","abstract":[{"lang":"eng","text":"The rising footprint of machine learning has led to a focus on imposing model\r\nsparsity as a means of reducing computational and memory costs. For deep neural\r\nnetworks (DNNs), the state-of-the-art accuracy-vs-sparsity is achieved by heuristics\r\ninspired by the classical Optimal Brain Surgeon (OBS) framework [LeCun et al.,\r\n1989, Hassibi and Stork, 1992, Hassibi et al., 1993], which leverages loss curvature\r\ninformation to make better pruning decisions. Yet, these results still lack a solid\r\ntheoretical understanding, and it is unclear whether they can be improved by\r\nleveraging connections to the wealth of work on sparse recovery algorithms. In this\r\npaper, we draw new connections between these two areas and present new sparse\r\nrecovery algorithms inspired by the OBS framework that comes with theoretical\r\nguarantees under reasonable assumptions and have strong practical performance.\r\nSpecifically, our work starts from the observation that we can leverage curvature\r\ninformation in OBS-like fashion upon the projection step of classic iterative sparse\r\nrecovery algorithms such as IHT. We show for the first time that this leads both\r\nto improved convergence bounds under standard assumptions. Furthermore, we\r\npresent extensions of this approach to the practical task of obtaining accurate sparse\r\nDNNs, and validate it experimentally at scale for Transformer-based models on\r\nvision and language tasks."}],"_id":"19518","article_processing_charge":"No","oa_version":"Preprint","citation":{"ieee":"D. Wu, I.-V. Modoranu, M. Safaryan, D. Kuznedelev, and D.-A. Alistarh, “The iterative optimal brain surgeon: Faster sparse recovery by leveraging second-order information,” in <i>38th Conference on Neural Information Processing Systems</i>, Vancouver, Canada, 2024, vol. 37.","apa":"Wu, D., Modoranu, I.-V., Safaryan, M., Kuznedelev, D., &#38; Alistarh, D.-A. (2024). The iterative optimal brain surgeon: Faster sparse recovery by leveraging second-order information. In <i>38th Conference on Neural Information Processing Systems</i> (Vol. 37). Vancouver, Canada: Neural Information Processing Systems Foundation.","ista":"Wu D, Modoranu I-V, Safaryan M, Kuznedelev D, Alistarh D-A. 2024. The iterative optimal brain surgeon: Faster sparse recovery by leveraging second-order information. 38th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 37.","short":"D. Wu, I.-V. Modoranu, M. Safaryan, D. Kuznedelev, D.-A. Alistarh, in:, 38th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2024.","chicago":"Wu, Diyuan, Ionut-Vlad Modoranu, Mher Safaryan, Denis Kuznedelev, and Dan-Adrian Alistarh. “The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information.” In <i>38th Conference on Neural Information Processing Systems</i>, Vol. 37. Neural Information Processing Systems Foundation, 2024.","mla":"Wu, Diyuan, et al. “The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information.” <i>38th Conference on Neural Information Processing Systems</i>, vol. 37, Neural Information Processing Systems Foundation, 2024.","ama":"Wu D, Modoranu I-V, Safaryan M, Kuznedelev D, Alistarh D-A. The iterative optimal brain surgeon: Faster sparse recovery by leveraging second-order information. In: <i>38th Conference on Neural Information Processing Systems</i>. Vol 37. Neural Information Processing Systems Foundation; 2024."},"date_updated":"2025-05-14T11:37:10Z","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","acknowledgement":"The authors thank the anonymous NeurIPS reviewers for their useful comments and feedback, the IT department from the Institute of Science and Technology Austria for the hardware support, and Weights and Biases for the infrastructure to track all our experiments. Mher Safaryan has received funding from the European Union’s Horizon 2020 research and innovation program under the Maria Skłodowska-Curie grant agreement No 101034413.","oa":1}]