---
OA_place: publisher
OA_type: gold
_id: '19519'
abstract:
- lang: eng
  text: There has been significant interest in "extreme" compression of large language
    models (LLMs), i.e. to 1-2 bits per parameter, which allows such models to be
    executed efficiently on resource-constrained devices. Existing work focused on
    improved one-shot quantization techniques and weight representations; yet, purely
    post-training approaches are reaching diminishing returns in terms of the accuracy-vs-bit-width
    trade-off. State-of-the-art quantization methods such as QuIP# and AQLM include
    fine-tuning (part of) the compressed parameters over a limited amount of calibration
    data; however, such fine-tuning techniques over compressed weights often make
    exclusive use of straight-through estimators (STE), whose performance is not well-understood
    in this setting. In this work, we question the use of STE for extreme LLM compression,
    showing that it can be sub-optimal, and perform a systematic study of quantization-aware
    fine-tuning strategies for LLMs.We propose PV-Tuning - a representation-agnostic
    framework that generalizes and improves upon existing fine-tuning strategies,
    and provides convergence guarantees in restricted cases.On the practical side,
    when used for 1-2 bit vector quantization, PV-Tuning outperforms prior techniques
    for highly-performant models such as Llama and Mistral. Using PV-Tuning, we achieve
    the first Pareto-optimal quantization for Llama-2 family models at 2 bits per
    parameter.
acknowledgement: "Authors would like to thank Vage Egiazarian, Andrei Panferov and
  Ruslan Svirschevski for their\r\nhelp and advice on AQLM codebase and running large-scale
  experiments. We also thank Philip\r\nZmushko and Artem Fedorov for helpful discussions
  during the early stages of our research. The research of Kai Yi, Konstantin Burlachenko,
  and Peter Richtárik reported in this publication was supported by funding from King
  Abdullah University of Science and Technology (KAUST) – Center of Excellence for
  Generative AI, under award number 5940. We would also like to thank our NeurIPS
  reviewers for their helpful suggestions, we specifically highlight p3Lv’s suggestions
  to consider smaller codebook sizes and evaluate PV-Tuning with QuIP#, both of which
  produced interesting findings. Finally, we thank the open-source contributors from
  llama.cpp9 and the LocalLlama10 community for discussions and inspirations on practical
  use cases of quantized language models, and in particular, Yalda Shabanzadeh and
  Arthur Aardvark for their help with improving the codebase."
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
arxiv: 1
author:
- first_name: Vladimir
  full_name: Malinovskii, Vladimir
  last_name: Malinovskii
- first_name: Denis
  full_name: Mazur, Denis
  last_name: Mazur
- first_name: Ivan
  full_name: Ilin, Ivan
  last_name: Ilin
- first_name: Denis
  full_name: Kuznedelev, Denis
  last_name: Kuznedelev
- first_name: Konstantin
  full_name: Burlachenko, Konstantin
  last_name: Burlachenko
- first_name: Kai
  full_name: Yi, Kai
  last_name: Yi
- first_name: Dan-Adrian
  full_name: Alistarh, Dan-Adrian
  id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
  last_name: Alistarh
  orcid: 0000-0003-3650-940X
- first_name: Peter
  full_name: Richtarik, Peter
  last_name: Richtarik
citation:
  ama: 'Malinovskii V, Mazur D, Ilin I, et al. PV-tuning: Beyond straight-through
    estimation for extreme LLM compression. In: <i>38th Conference on Neural Information
    Processing Systems</i>. Vol 37. Neural Information Processing Systems Foundation;
    2024.'
  apa: 'Malinovskii, V., Mazur, D., Ilin, I., Kuznedelev, D., Burlachenko, K., Yi,
    K., … Richtarik, P. (2024). PV-tuning: Beyond straight-through estimation for
    extreme LLM compression. In <i>38th Conference on Neural Information Processing
    Systems</i> (Vol. 37). Vancouver, Canada: Neural Information Processing Systems
    Foundation.'
  chicago: 'Malinovskii, Vladimir, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin
    Burlachenko, Kai Yi, Dan-Adrian Alistarh, and Peter Richtarik. “PV-Tuning: Beyond
    Straight-through Estimation for Extreme LLM Compression.” In <i>38th Conference
    on Neural Information Processing Systems</i>, Vol. 37. Neural Information Processing
    Systems Foundation, 2024.'
  ieee: 'V. Malinovskii <i>et al.</i>, “PV-tuning: Beyond straight-through estimation
    for extreme LLM compression,” in <i>38th Conference on Neural Information Processing
    Systems</i>, Vancouver, Canada, 2024, vol. 37.'
  ista: 'Malinovskii V, Mazur D, Ilin I, Kuznedelev D, Burlachenko K, Yi K, Alistarh
    D-A, Richtarik P. 2024. PV-tuning: Beyond straight-through estimation for extreme
    LLM compression. 38th Conference on Neural Information Processing Systems. NeurIPS:
    Neural Information Processing Systems, Advances in Neural Information Processing
    Systems, vol. 37.'
  mla: 'Malinovskii, Vladimir, et al. “PV-Tuning: Beyond Straight-through Estimation
    for Extreme LLM Compression.” <i>38th Conference on Neural Information Processing
    Systems</i>, vol. 37, Neural Information Processing Systems Foundation, 2024.'
  short: V. Malinovskii, D. Mazur, I. Ilin, D. Kuznedelev, K. Burlachenko, K. Yi,
    D.-A. Alistarh, P. Richtarik, in:, 38th Conference on Neural Information Processing
    Systems, Neural Information Processing Systems Foundation, 2024.
conference:
  end_date: 2024-12-15
  location: Vancouver, Canada
  name: 'NeurIPS: Neural Information Processing Systems'
  start_date: 2024-12-10
date_created: 2025-04-06T22:01:32Z
date_published: 2024-12-20T00:00:00Z
date_updated: 2025-05-14T10:49:20Z
day: '20'
ddc:
- '000'
department:
- _id: DaAl
external_id:
  arxiv:
  - '2405.14852'
file:
- access_level: open_access
  checksum: 54d36f947887e26d0e568b512167001a
  content_type: application/pdf
  creator: dernst
  date_created: 2025-04-07T09:17:10Z
  date_updated: 2025-04-07T09:17:10Z
  file_id: '19521'
  file_name: 2024_NeurIPS_Malinovskii.pdf
  file_size: 939712
  relation: main_file
  success: 1
file_date_updated: 2025-04-07T09:17:10Z
has_accepted_license: '1'
intvolume: '        37'
language:
- iso: eng
month: '12'
oa: 1
oa_version: Published Version
publication: 38th Conference on Neural Information Processing Systems
publication_identifier:
  isbn:
  - '9798331314385'
  issn:
  - 1049-5258
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'PV-tuning: Beyond straight-through estimation for extreme LLM compression'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 37
year: '2024'
...