{"oa":1,"tmp":{"legal_code_url":"https://creativecommons.org/licenses/by/4.0/legalcode","name":"Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)","short":"CC BY (4.0)","image":"/images/cc_by.png"},"date_updated":"2026-06-03T05:53:30Z","corr_author":"1","quality_controlled":"1","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","OA_place":"publisher","_id":"21932","citation":{"short":"E. Schultheis, D.-A. Alistarh, in:, 2nd Conference on Parsimony and Learning, ML Research Press, 2026, pp. 265–284.","ieee":"E. Schultheis and D.-A. Alistarh, “LLMQ: Efficient lower-precision LLM training for consumer GPUs,” in <i>2nd Conference on Parsimony and Learning</i>, Stanford, CA, United States, 2026, vol. 328, pp. 265–284.","chicago":"Schultheis, Erik, and Dan-Adrian Alistarh. “LLMQ: Efficient Lower-Precision LLM Training for Consumer GPUs.” In <i>2nd Conference on Parsimony and Learning</i>, 328:265–84. ML Research Press, 2026.","ama":"Schultheis E, Alistarh D-A. LLMQ: Efficient lower-precision LLM training for consumer GPUs. In: <i>2nd Conference on Parsimony and Learning</i>. Vol 328. ML Research Press; 2026:265-284.","ista":"Schultheis E, Alistarh D-A. 2026. LLMQ: Efficient lower-precision LLM training for consumer GPUs. 2nd Conference on Parsimony and Learning. CPAL: Conference on Parsimony and Learning, PMLR, vol. 328, 265–284.","apa":"Schultheis, E., &#38; Alistarh, D.-A. (2026). LLMQ: Efficient lower-precision LLM training for consumer GPUs. In <i>2nd Conference on Parsimony and Learning</i> (Vol. 328, pp. 265–284). Stanford, CA, United States: ML Research Press.","mla":"Schultheis, Erik, and Dan-Adrian Alistarh. “LLMQ: Efficient Lower-Precision LLM Training for Consumer GPUs.” <i>2nd Conference on Parsimony and Learning</i>, vol. 328, ML Research Press, 2026, pp. 265–84."},"acknowledgement":"We would like to thank contacts at NVIDIA (Vartika Singh, Nina Carrejo, Kyla Wilkes, and Tijmen\r\nBlankevoort), HP (Curtis Burkhalter), and Datacrunch/Verda (Paul Chang and Antonio\r\nDominguez) for hardware support that was essential to this project. ES was supported in part\r\nby ERC Proof-of-Concept grant FastML.","author":[{"full_name":"Schultheis, Erik","first_name":"Erik","last_name":"Schultheis","id":"2786b299-e6b0-11f0-91da-9243fe3ef96b"},{"last_name":"Alistarh","first_name":"Dan-Adrian","full_name":"Alistarh, Dan-Adrian","orcid":"0000-0003-3650-940X","id":"4A899BFC-F248-11E8-B48F-1D18A9856A87"}],"publication_identifier":{"eissn":["2640-3498"]},"ddc":["000"],"oa_version":"Published Version","date_created":"2026-05-31T22:02:13Z","month":"04","language":[{"iso":"eng"}],"related_material":{"link":[{"relation":"software","url":"https://github.com/IST-DASLab/llmq"}]},"date_published":"2026-04-06T00:00:00Z","publisher":"ML Research Press","file_date_updated":"2026-06-03T05:51:19Z","conference":{"end_date":"2025-03-27","location":"Stanford, CA, United States","start_date":"2025-03-24","name":"CPAL: Conference on Parsimony and Learning"},"type":"conference","intvolume":"       328","status":"public","file":[{"date_updated":"2026-06-03T05:51:19Z","relation":"main_file","file_size":2099944,"checksum":"72f9a87c70f1e2105ef64050ee5017e5","date_created":"2026-06-03T05:51:19Z","file_name":"2026_CPAL_Schultheis.pdf","content_type":"application/pdf","success":1,"access_level":"open_access","file_id":"21942","creator":"dernst"}],"abstract":[{"lang":"eng","text":"We present LLMQ, an end-to-end CUDA/C++ implementation for medium-sized language-model training, e.g. 3B to 32B parameters, on affordable, commodity GPUs. These devices are characterized by low memory availability and slow communication compared to datacentre-grade GPUs. Consequently, we showcase a range of optimizations that target these bottlenecks, including activation checkpointing, offloading, and copy-engine based collectives. LLMQ is able to train or fine-tune a 7B model on a single 16GB mid-range gaming card, or a 32B model on a workstation equipped with 4 RTX 4090s. This is achieved while executing a standard 8-bit training pipeline, without additional algorithmic approximations, and maintaining FLOP utilization of around 50%. The efficiency of LLMQ rivals that of production-scale systems on much more expensive cloud-grade GPUs."}],"publication_status":"published","page":"265-284","article_processing_charge":"No","publication":"2nd Conference on Parsimony and Learning","day":"06","title":"LLMQ: Efficient lower-precision LLM training for consumer GPUs","has_accepted_license":"1","year":"2026","scopus_import":"1","OA_type":"diamond","alternative_title":["PMLR"],"department":[{"_id":"DaAl"}],"volume":328}