{"month":"05","abstract":[{"text":"Generative Pre-trained Transformer models, known as GPT or OPT, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models may require multiple performant GPUs, which limits the usability of such models. While there is emerging work on relieving this pressure via model compression, the applicability and performance of existing compression techniques is limited by the scale and complexity of GPT models. In this paper, we address this challenge, and propose OPTQ, a new one-shot weight quantization method based on approximate second-order information, that is both highly-accurate and highly-efficient. Specifically, OPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first time to execute an 175 billion-parameter model inside a single GPU for generative inference. Moreover, we also show that our method can still provide reasonable accuracy in the extreme quantization regime, in which weights are quantized to 2-bit or even ternary quantization levels. We show experimentally that these improvements can be leveraged for end-to-end inference speedups over FP16, of around 3.25x when using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000). The implementation is available at https://github.com/IST-DASLab/gptq.","lang":"eng"}],"scopus_import":"1","date_updated":"2024-08-05T07:55:42Z","year":"2023","author":[{"first_name":"Elias","full_name":"Frantar, Elias","id":"09a8f98d-ec99-11ea-ae11-c063a7b7fe5f","last_name":"Frantar"},{"full_name":"Ashkboos, Saleh","last_name":"Ashkboos","first_name":"Saleh"},{"last_name":"Hoefler","full_name":"Hoefler, Torsten","first_name":"Torsten"},{"orcid":"0000-0003-3650-940X","first_name":"Dan-Adrian","last_name":"Alistarh","id":"4A899BFC-F248-11E8-B48F-1D18A9856A87","full_name":"Alistarh, Dan-Adrian"}],"citation":{"ama":"Frantar E, Ashkboos S, Hoefler T, Alistarh D-A. OPTQ: Accurate post-training quantization for generative pre-trained transformers. In: <i>11th International Conference on Learning Representations </i>. International Conference on Learning Representations; 2023.","chicago":"Frantar, Elias, Saleh Ashkboos, Torsten Hoefler, and Dan-Adrian Alistarh. “OPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers.” In <i>11th International Conference on Learning Representations </i>. International Conference on Learning Representations, 2023.","short":"E. Frantar, S. Ashkboos, T. Hoefler, D.-A. Alistarh, in:, 11th International Conference on Learning Representations , International Conference on Learning Representations, 2023.","apa":"Frantar, E., Ashkboos, S., Hoefler, T., &#38; Alistarh, D.-A. (2023). OPTQ: Accurate post-training quantization for generative pre-trained transformers. In <i>11th International Conference on Learning Representations </i>. Kigali, Rwanda: International Conference on Learning Representations.","ista":"Frantar E, Ashkboos S, Hoefler T, Alistarh D-A. 2023. OPTQ: Accurate post-training quantization for generative pre-trained transformers. 11th International Conference on Learning Representations . ICLR: International Conference on Learning Representations.","ieee":"E. Frantar, S. Ashkboos, T. Hoefler, and D.-A. Alistarh, “OPTQ: Accurate post-training quantization for generative pre-trained transformers,” in <i>11th International Conference on Learning Representations </i>, Kigali, Rwanda, 2023.","mla":"Frantar, Elias, et al. “OPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers.” <i>11th International Conference on Learning Representations </i>, International Conference on Learning Representations, 2023."},"type":"conference","_id":"17378","date_published":"2023-05-01T00:00:00Z","status":"public","date_created":"2024-08-04T22:01:22Z","publication_status":"published","ddc":["000"],"oa_version":"Published Version","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","publisher":"International Conference on Learning Representations","has_accepted_license":"1","title":"OPTQ: Accurate post-training quantization for generative pre-trained transformers","related_material":{"link":[{"relation":"software","url":"https://github.com/IST-DASLab/gptq"}]},"oa":1,"conference":{"name":"ICLR: International Conference on Learning Representations","location":"Kigali, Rwanda","end_date":"2023-05-05","start_date":"2023-05-01"},"file":[{"success":1,"file_id":"17385","creator":"dernst","relation":"main_file","file_name":"2023_ICLR_Frantar.pdf","file_size":437492,"date_updated":"2024-08-05T07:52:44Z","date_created":"2024-08-05T07:52:44Z","checksum":"aacbf11dbd8b02a3e0bfd942a33e0593","content_type":"application/pdf","access_level":"open_access"}],"publication":"11th International Conference on Learning Representations ","department":[{"_id":"DaAl"}],"ec_funded":1,"language":[{"iso":"eng"}],"acknowledged_ssus":[{"_id":"ScienComp"}],"corr_author":"1","day":"01","quality_controlled":"1","acknowledgement":"Elias Frantar and Dan Alistarh gratefully acknowledge funding from the European Research Council (ERC) under the European Union’s Horizon 2020 programme (grant agreement No. 805223 ScaleML), as well as experimental support from Eldar Kurtic, and from the IST Austria IT department, in particular Stefano Elefante, Andrei Hornoiu, and Alois Schloegl. The work of Saleh Ashkboos and Torsten Hoefler was supported by the PASC DaCeMI project, received EuroHPC-JU funding under grant MAELSTROM, No. 955513. We thank the Swiss National Supercomputing Center (CSCS) for supporting us with compute infrastructure.","article_processing_charge":"No","project":[{"grant_number":"805223","name":"Elastic Coordination for Scalable Machine Learning","call_identifier":"H2020","_id":"268A44D6-B435-11E9-9278-68D0E5697425"}],"file_date_updated":"2024-08-05T07:52:44Z"}