---
OA_place: publisher
OA_type: green
_id: '21857'
abstract:
- lang: eng
  text: "The availability of powerful open-source large language models (LLMs) opens
    exciting use cases, such as using personal data to fine-tune these models to imitate
    a user’s unique writing style. Two key requirements for this functionality are
    personalization–in the sense that the output should recognizably reflect the user’s
    own writing style—and privacy–users may justifiably be wary of uploading extremely
    personal data, such as their email archive, to a third-party service. In this
    paper, we demonstrate the feasibility of training and running such an assistant,
    which we call Panza, on commodity hardware, for the specific use case of email
    generation. Panza’s personalization features are based on a combination of parameter-efficient
    fine-tuning using a variant of the Reverse Instructions technique [1] and Retrieval-Augmented
    Generation (RAG) [2]. We demonstrate that this combination allows us to fine-tune
    an LLM to reflect a user’s writing style using limited data, while executing on
    extremely limited resources, e.g. on a free Google Colab instance. Our key methodological
    contribution is the first detailed study of evaluation metrics for this task,
    and\r\nof how different choices of system components–the use of RAG and of different
    fine-tuning approaches–impact the system’s performance. Additionally, we demonstrate
    that very little data - under 100 email samples - are sufficient to create models
    that convincingly imitate humans, showcasing a previously unknown attack vector
    in language models. We are releasing the full Panza code as well as three new
    email datasets licensed for research use."
article_number: '81'
article_processing_charge: No
author:
- first_name: Armand
  full_name: Nicolicioiu, Armand
  last_name: Nicolicioiu
- first_name: Eugenia B
  full_name: Iofinova, Eugenia B
  id: f9a17499-f6e0-11ea-865d-fdf9a3f77117
  last_name: Iofinova
  orcid: 0000-0002-7778-3221
- first_name: Andrej
  full_name: Jovanovic, Andrej
  last_name: Jovanovic
- first_name: Eldar
  full_name: Kurtic, Eldar
  id: 47beb3a5-07b5-11eb-9b87-b108ec578218
  last_name: Kurtic
- first_name: Mahdi
  full_name: Nikdan, Mahdi
  id: 66374281-f394-11eb-9cf6-869147deecc0
  last_name: Nikdan
- first_name: Andrei
  full_name: Panferov, Andrei
  id: 2c18daae-4dbe-11ef-8491-98ce2d960f09
  last_name: Panferov
- first_name: Ilia
  full_name: Markov, Ilia
  id: D0CF4148-C985-11E9-8066-0BDEE5697425
  last_name: Markov
- first_name: Nir
  full_name: Shavit, Nir
  last_name: Shavit
- first_name: Dan-Adrian
  full_name: Alistarh, Dan-Adrian
  id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
  last_name: Alistarh
  orcid: 0000-0003-3650-940X
citation:
  ama: 'Nicolicioiu A, Iofinova EB, Jovanovic A, et al. <i>Panza: Investigating the
    Feasibility of Fully-Local Personalized Text Generation</i>. OpenReview; 2026.'
  apa: 'Nicolicioiu, A., Iofinova, E. B., Jovanovic, A., Kurtic, E., Nikdan, M., Panferov,
    A., … Alistarh, D.-A. (2026). <i>Panza: Investigating the feasibility of fully-local
    personalized text generation</i>. <i>Third Conference on Parsimony and Learning
    (Proceedings Track)</i>. Tübíngen, Germany: OpenReview.'
  chicago: 'Nicolicioiu, Armand, Eugenia B Iofinova, Andrej Jovanovic, Eldar Kurtic,
    Mahdi Nikdan, Andrei Panferov, Ilia Markov, Nir Shavit, and Dan-Adrian Alistarh.
    <i>Panza: Investigating the Feasibility of Fully-Local Personalized Text Generation</i>.
    <i>Third Conference on Parsimony and Learning (Proceedings Track)</i>. OpenReview,
    2026.'
  ieee: 'A. Nicolicioiu <i>et al.</i>, <i>Panza: Investigating the feasibility of
    fully-local personalized text generation</i>. OpenReview, 2026.'
  ista: 'Nicolicioiu A, Iofinova EB, Jovanovic A, Kurtic E, Nikdan M, Panferov A,
    Markov I, Shavit N, Alistarh D-A. 2026. Panza: Investigating the feasibility of
    fully-local personalized text generation, OpenReview,p.'
  mla: 'Nicolicioiu, Armand, et al. “Panza: Investigating the Feasibility of Fully-Local
    Personalized Text Generation.” <i>Third Conference on Parsimony and Learning (Proceedings
    Track)</i>, 81, OpenReview, 2026.'
  short: 'A. Nicolicioiu, E.B. Iofinova, A. Jovanovic, E. Kurtic, M. Nikdan, A. Panferov,
    I. Markov, N. Shavit, D.-A. Alistarh, Panza: Investigating the Feasibility of
    Fully-Local Personalized Text Generation, OpenReview, 2026.'
conference:
  end_date: 2026-03-26
  location: Tübíngen, Germany
  name: 'CPAL: Conference on Parsimony and Learning'
  start_date: 2026-03-23
corr_author: '1'
date_created: 2026-05-11T08:50:28Z
date_published: 2026-03-06T00:00:00Z
date_updated: 2026-05-19T11:20:27Z
day: '06'
department:
- _id: GradSch
- _id: DaAl
keyword:
- LLMs
- PEFT
- LoRA
- personalization
- efficient ML
language:
- iso: eng
license: https://creativecommons.org/licenses/by/4.0/
main_file_link:
- open_access: '1'
  url: https://openreview.net/pdf?id=soFWnTqd23
month: '03'
oa: 1
oa_version: Accepted Version
publication: Third Conference on Parsimony and Learning (Proceedings Track)
publication_status: published
publisher: OpenReview
quality_controlled: '1'
related_material:
  record:
  - id: '21854'
    relation: dissertation_contains
    status: public
status: public
title: 'Panza: Investigating the feasibility of fully-local personalized text generation'
tmp:
  image: /images/cc_by.png
  legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
  name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
  short: CC BY (4.0)
type: conference_poster
user_id: 8b945eb4-e2f2-11eb-945a-df72226e66a9
year: '2026'
...
---
_id: '18113'
abstract:
- lang: eng
  text: 'The emergence of accurate open large language models (LLMs) has led to a
    race towards performant quantization techniques which can enable their execution
    on end-user devices. In this paper, we revisit the problem of “extreme” LLM compression—defined
    as targeting extremely low bit counts, such as 2 to 3 bits per parameter—from
    the point of view of classic methods in Multi-Codebook Quantization (MCQ). Our
    algorithm, called AQLM, generalizes the classic Additive Quantization (AQ) approach
    for information retrieval to advance the state-of-the-art in LLM compression,
    via two innovations: 1) learned additive quantization of weight matrices in input-adaptive
    fashion, and 2) joint optimization of codebook parameters across each transformer
    blocks. Broadly, AQLM is the first scheme that is Pareto optimal in terms of accuracy-vs-model-size
    when compressing to less than 3 bits per parameter, and significantly improves
    upon all known schemes in the extreme compression (2bit) regime. In addition,
    AQLM is practical: we provide fast GPU and CPU implementations of AQLM for token
    generation, which enable us to match or outperform optimized FP16 implementations
    for speed, while executing in a much smaller memory footprint.'
acknowledgement: "Authors would like to thank Ruslan Svirschevski for his help in
  solving technical issues with AQLM and baselines. We also thank Tim Dettmers for
  helpful discussions on the structure of weights in modern LLMs and size-accuracy
  trade-offs. The authors would also like to thank Daniil Pavlov for his assistance
  with CPU benchmarking. Finally, authors would like to thank the communities of ML
  enthusiasts known as LocalLLaMA5 and Petals community on discord6\r\nfor the crowd
  wisdom about running LLMs on consumer devices. Egiazarian Vage and Denis Kuznedelev
  and Andrei Panferov were supported by the grant for research centers in the field
  of AI provided by the Analytical Center for the Government of the Russian Federation
  (ACRF) in\r\naccordance with the agreement on the provision of subsidies (identifier
  of the agreement 000000D730321P5Q0002) and the agreement with HSE University No.
  70-2021-00139."
alternative_title:
- PMLR
article_processing_charge: No
arxiv: 1
author:
- first_name: Vage
  full_name: Egiazarian, Vage
  last_name: Egiazarian
- first_name: Andrei
  full_name: Panferov, Andrei
  id: 2c18daae-4dbe-11ef-8491-98ce2d960f09
  last_name: Panferov
- first_name: Denis
  full_name: Kuznedelev, Denis
  last_name: Kuznedelev
- first_name: Elias
  full_name: Frantar, Elias
  id: 09a8f98d-ec99-11ea-ae11-c063a7b7fe5f
  last_name: Frantar
- first_name: Artem
  full_name: Babenko, Artem
  last_name: Babenko
- first_name: Dan-Adrian
  full_name: Alistarh, Dan-Adrian
  id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
  last_name: Alistarh
  orcid: 0000-0003-3650-940X
citation:
  ama: 'Egiazarian V, Panferov A, Kuznedelev D, Frantar E, Babenko A, Alistarh D-A.
    Extreme compression of large language models via additive quantization. In: <i>Proceedings
    of the 41st International Conference on Machine Learning</i>. Vol 235. ML Research
    Press; 2024:12284-12303.'
  apa: 'Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., &#38;
    Alistarh, D.-A. (2024). Extreme compression of large language models via additive
    quantization. In <i>Proceedings of the 41st International Conference on Machine
    Learning</i> (Vol. 235, pp. 12284–12303). Vienna, Austria: ML Research Press.'
  chicago: Egiazarian, Vage, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem
    Babenko, and Dan-Adrian Alistarh. “Extreme Compression of Large Language Models
    via Additive Quantization.” In <i>Proceedings of the 41st International Conference
    on Machine Learning</i>, 235:12284–303. ML Research Press, 2024.
  ieee: V. Egiazarian, A. Panferov, D. Kuznedelev, E. Frantar, A. Babenko, and D.-A.
    Alistarh, “Extreme compression of large language models via additive quantization,”
    in <i>Proceedings of the 41st International Conference on Machine Learning</i>,
    Vienna, Austria, 2024, vol. 235, pp. 12284–12303.
  ista: 'Egiazarian V, Panferov A, Kuznedelev D, Frantar E, Babenko A, Alistarh D-A.
    2024. Extreme compression of large language models via additive quantization.
    Proceedings of the 41st International Conference on Machine Learning. ICML: International
    Conference on Machine Learning, PMLR, vol. 235, 12284–12303.'
  mla: Egiazarian, Vage, et al. “Extreme Compression of Large Language Models via
    Additive Quantization.” <i>Proceedings of the 41st International Conference on
    Machine Learning</i>, vol. 235, ML Research Press, 2024, pp. 12284–303.
  short: V. Egiazarian, A. Panferov, D. Kuznedelev, E. Frantar, A. Babenko, D.-A.
    Alistarh, in:, Proceedings of the 41st International Conference on Machine Learning,
    ML Research Press, 2024, pp. 12284–12303.
conference:
  end_date: 2024-07-27
  location: Vienna, Austria
  name: 'ICML: International Conference on Machine Learning'
  start_date: 2024-07-21
corr_author: '1'
date_created: 2024-09-22T22:01:43Z
date_published: 2024-09-01T00:00:00Z
date_updated: 2024-10-01T08:13:05Z
day: '01'
department:
- _id: DaAl
- _id: GradSch
external_id:
  arxiv:
  - '2401.06118'
intvolume: '       235'
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: ' https://doi.org/10.48550/arXiv.2401.06118'
month: '09'
oa: 1
oa_version: Preprint
page: 12284-12303
publication: Proceedings of the 41st International Conference on Machine Learning
publication_identifier:
  eissn:
  - 2640-3498
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
scopus_import: '1'
status: public
title: Extreme compression of large language models via additive quantization
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 235
year: '2024'
...