{"month":"12","main_file_link":[{"open_access":"1","url":"https://doi.org/10.48550/arXiv.2501.02625"}],"article_processing_charge":"No","supplementarymaterial":"no","OA_type":"green","OA_place":"repository","quality_controlled":"1","arxiv":1,"department":[{"_id":"DaAl"},{"_id":"GradSch"}],"oa_version":"Preprint","oa":1,"date_created":"2026-09-06T22:02:00Z","author":[{"last_name":"Ashkboos","full_name":"Ashkboos, Saleh","first_name":"Saleh"},{"id":"66374281-f394-11eb-9cf6-869147deecc0","full_name":"Nikdan, Mahdi","last_name":"Nikdan","first_name":"Mahdi"},{"first_name":"Soroush","full_name":"Tabesh, Soroush","last_name":"Tabesh","id":"06000900-6068-11ef-8d61-c2472ef2e752","orcid":"0009-0003-4119-6281"},{"first_name":"Roberto","full_name":"Lopez Castro, Roberto","last_name":"Lopez Castro","id":"18495c31-fb57-11ef-ba0d-c290e10e394e"},{"last_name":"Hoefler","full_name":"Hoefler, Torsten","first_name":"Torsten"},{"orcid":"0000-0003-3650-940X","id":"4A899BFC-F248-11E8-B48F-1D18A9856A87","full_name":"Alistarh, Dan-Adrian","last_name":"Alistarh","first_name":"Dan-Adrian"}],"scopus_import":"1","acknowledgement":"This project has received funding from the European Research Council (ERC) under the European\r\nUnion’s Horizon 2020 program (grant agreement PSAP, No. 101002047. This research also obtained\r\nfunding from the “UrbanTwin: An urban digital twin for climate action: Assessing policies and\r\nsolutions for energy, water and infrastructure” project, funded by the ETH-Domain Joint Initiative\r\nprogram in the Strategic Area Energy, Climate and Sustainable Environment.","status":"public","fulldoi":"https://doi.org/10.52202/085713-3966","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","date_published":"2025-12-01T00:00:00Z","year":"2025","_id":"22827","language":[{"iso":"eng"}],"page":"131755-131780","publication":"39th Conference on Neural Information Processing Systems","title":"HALO: Hadamard-assisted lower-precision optimization for LLMs","external_id":{"arxiv":["2501.02625"]},"volume":38,"alternative_title":["Advances in Neural Information Processing Systems"],"publisher":"Neural Information Processing Systems Foundation","type":"conference","day":"01","publication_status":"published","publication_identifier":{"isbn":["9798331338275"],"eissn":["1049-5258"]},"das_tickbox":"0","abstract":[{"lang":"eng","text":"Quantized training of Large Language Models (LLMs) remains an open challenge, as maintaining accuracy while performing all matrix multiplications in low precision has proven difficult. This is particularly the case when fine-tuning pre-trained models, which can have large weight, activation, and error (output gradient) outlier values that make lower-precision optimization difficult. To address this, we present HALO, a new quantization-aware training approach for Transformers that enables accurate and efficient low-precision training by combining 1) strategic placement of Hadamard rotations in both forward and backward passes, which mitigate outliers, 2) high-performance kernel support, and 3) FSDP integration for low-precision communication. Our approach ensures that all large matrix multiplications during the forward and backward passes are executed in lower precision. Applied to LLaMa models, HALO achieves near-full-precision-equivalent results during fine-tuning on various tasks, while delivering up to 1.41x end-to-end speedup for full fine-tuning on RTX 4090 GPUs. HALO efficiently supports both standard and parameter-efficient fine-tuning (PEFT). Our results demonstrate the first practical approach to fully quantized LLM fine-tuning that maintains accuracy in INT8 and FP6 precision, while delivering performance benefits."}],"citation":{"ama":"Ashkboos S, Nikdan M, Tabesh S, Lopez Castro R, Hoefler T, Alistarh D-A. HALO: Hadamard-assisted lower-precision optimization for LLMs. In: 39th Conference on Neural Information Processing Systems. Vol 38. Neural Information Processing Systems Foundation; 2025:131755-131780. doi:10.52202/085713-3966","short":"S. Ashkboos, M. Nikdan, S. Tabesh, R. Lopez Castro, T. Hoefler, D.-A. Alistarh, in:, 39th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2025, pp. 131755–131780.","ista":"Ashkboos S, Nikdan M, Tabesh S, Lopez Castro R, Hoefler T, Alistarh D-A. 2025. HALO: Hadamard-assisted lower-precision optimization for LLMs. 39th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 38, 131755–131780.","chicago":"Ashkboos, Saleh, Mahdi Nikdan, Soroush Tabesh, Roberto Lopez Castro, Torsten Hoefler, and Dan-Adrian Alistarh. “HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs.” In 39th Conference on Neural Information Processing Systems, 38:131755–80. Neural Information Processing Systems Foundation, 2025. https://doi.org/10.52202/085713-3966.","ieee":"S. Ashkboos, M. Nikdan, S. Tabesh, R. Lopez Castro, T. Hoefler, and D.-A. Alistarh, “HALO: Hadamard-assisted lower-precision optimization for LLMs,” in 39th Conference on Neural Information Processing Systems, San Diego, CA, United States, 2025, vol. 38, pp. 131755–131780.","mla":"Ashkboos, Saleh, et al. “HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs.” 39th Conference on Neural Information Processing Systems, vol. 38, Neural Information Processing Systems Foundation, 2025, pp. 131755–80, doi:10.52202/085713-3966.","apa":"Ashkboos, S., Nikdan, M., Tabesh, S., Lopez Castro, R., Hoefler, T., & Alistarh, D.-A. (2025). HALO: Hadamard-assisted lower-precision optimization for LLMs. In 39th Conference on Neural Information Processing Systems (Vol. 38, pp. 131755–131780). San Diego, CA, United States: Neural Information Processing Systems Foundation. https://doi.org/10.52202/085713-3966"},"intvolume":" 38","conference":{"location":"San Diego, CA, United States","name":"NeurIPS: Neural Information Processing Systems","end_date":"2025-12-07","start_date":"2025-12-02"},"researchdata_availability":"no","doi":"10.52202/085713-3966","date_updated":"2026-09-17T06:39:18Z"}