{"external_id":{"arxiv":["2404.00456"]},"intvolume":"        37","scopus_import":"1","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","month":"12","article_processing_charge":"No","OA_type":"green","year":"2024","date_created":"2025-04-06T22:01:32Z","date_published":"2024-12-20T00:00:00Z","publication":"38th Conference on Neural Information Processing Systems","publication_status":"published","day":"20","citation":{"short":"S. Ashkboos, A. Mohtashami, M.L. Croci, B. Li, P. Cameron, M. Jaggi, D.-A. Alistarh, T. Hoefler, J. Hensman, in:, 38th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2024.","apa":"Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., … Hensman, J. (2024). QuaRot: Outlier-free 4-bit inference in rotated LLMs. In <i>38th Conference on Neural Information Processing Systems</i> (Vol. 37). Vancouver, Canada: Neural Information Processing Systems Foundation.","mla":"Ashkboos, Saleh, et al. “QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs.” <i>38th Conference on Neural Information Processing Systems</i>, vol. 37, Neural Information Processing Systems Foundation, 2024.","ieee":"S. Ashkboos <i>et al.</i>, “QuaRot: Outlier-free 4-bit inference in rotated LLMs,” in <i>38th Conference on Neural Information Processing Systems</i>, Vancouver, Canada, 2024, vol. 37.","ista":"Ashkboos S, Mohtashami A, Croci ML, Li B, Cameron P, Jaggi M, Alistarh D-A, Hoefler T, Hensman J. 2024. QuaRot: Outlier-free 4-bit inference in rotated LLMs. 38th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 37.","chicago":"Ashkboos, Saleh, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan-Adrian Alistarh, Torsten Hoefler, and James Hensman. “QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs.” In <i>38th Conference on Neural Information Processing Systems</i>, Vol. 37. Neural Information Processing Systems Foundation, 2024.","ama":"Ashkboos S, Mohtashami A, Croci ML, et al. QuaRot: Outlier-free 4-bit inference in rotated LLMs. In: <i>38th Conference on Neural Information Processing Systems</i>. Vol 37. Neural Information Processing Systems Foundation; 2024."},"publisher":"Neural Information Processing Systems Foundation","publication_identifier":{"issn":["1049-5258"]},"title":"QuaRot: Outlier-free 4-bit inference in rotated LLMs","type":"conference","oa_version":"Preprint","alternative_title":["Advances in Neural Information Processing Systems"],"author":[{"first_name":"Saleh","last_name":"Ashkboos","full_name":"Ashkboos, Saleh"},{"full_name":"Mohtashami, Amirkeivan","first_name":"Amirkeivan","last_name":"Mohtashami"},{"first_name":"Maximilian L.","last_name":"Croci","full_name":"Croci, Maximilian L."},{"first_name":"Bo","last_name":"Li","full_name":"Li, Bo"},{"first_name":"Pashmina","last_name":"Cameron","full_name":"Cameron, Pashmina"},{"first_name":"Martin","last_name":"Jaggi","full_name":"Jaggi, Martin"},{"first_name":"Dan-Adrian","last_name":"Alistarh","full_name":"Alistarh, Dan-Adrian","id":"4A899BFC-F248-11E8-B48F-1D18A9856A87","orcid":"0000-0003-3650-940X"},{"full_name":"Hoefler, Torsten","last_name":"Hoefler","first_name":"Torsten"},{"first_name":"James","last_name":"Hensman","full_name":"Hensman, James"}],"oa":1,"abstract":[{"lang":"eng","text":"We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism, and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4 bits, without any channels identified for retention in higher precision. Our 4-bit quantized LLAMA2-70B model has losses of at most 0.47 WikiText-2 perplexity and retains 99% of the zero-shot performance. We also show that QuaRot can provide lossless 6 and 8 bit LLAMA-2 models without any calibration data using round-to-nearest quantization. Code is available at github.com/spcl/QuaRot."}],"quality_controlled":"1","related_material":{"link":[{"relation":"software","url":"https://github.com/spcl/QuaRot"}]},"conference":{"end_date":"2024-12-15","name":"NeurIPS: Neural Information Processing Systems","location":"Vancouver, Canada","start_date":"2024-12-09"},"_id":"19511","department":[{"_id":"DaAl"}],"main_file_link":[{"url":"https://doi.org/10.48550/arXiv.2404.00456","open_access":"1"}],"OA_place":"repository","volume":37,"status":"public","date_updated":"2025-05-14T11:33:12Z","language":[{"iso":"eng"}],"arxiv":1}