---
OA_place: publisher
OA_type: green
_id: '21857'
abstract:
- lang: eng
  text: "The availability of powerful open-source large language models (LLMs) opens
    exciting use cases, such as using personal data to fine-tune these models to imitate
    a user’s unique writing style. Two key requirements for this functionality are
    personalization–in the sense that the output should recognizably reflect the user’s
    own writing style—and privacy–users may justifiably be wary of uploading extremely
    personal data, such as their email archive, to a third-party service. In this
    paper, we demonstrate the feasibility of training and running such an assistant,
    which we call Panza, on commodity hardware, for the specific use case of email
    generation. Panza’s personalization features are based on a combination of parameter-efficient
    fine-tuning using a variant of the Reverse Instructions technique [1] and Retrieval-Augmented
    Generation (RAG) [2]. We demonstrate that this combination allows us to fine-tune
    an LLM to reflect a user’s writing style using limited data, while executing on
    extremely limited resources, e.g. on a free Google Colab instance. Our key methodological
    contribution is the first detailed study of evaluation metrics for this task,
    and\r\nof how different choices of system components–the use of RAG and of different
    fine-tuning approaches–impact the system’s performance. Additionally, we demonstrate
    that very little data - under 100 email samples - are sufficient to create models
    that convincingly imitate humans, showcasing a previously unknown attack vector
    in language models. We are releasing the full Panza code as well as three new
    email datasets licensed for research use."
article_number: '81'
article_processing_charge: No
author:
- first_name: Armand
  full_name: Nicolicioiu, Armand
  last_name: Nicolicioiu
- first_name: Eugenia B
  full_name: Iofinova, Eugenia B
  id: f9a17499-f6e0-11ea-865d-fdf9a3f77117
  last_name: Iofinova
  orcid: 0000-0002-7778-3221
- first_name: Andrej
  full_name: Jovanovic, Andrej
  last_name: Jovanovic
- first_name: Eldar
  full_name: Kurtic, Eldar
  id: 47beb3a5-07b5-11eb-9b87-b108ec578218
  last_name: Kurtic
- first_name: Mahdi
  full_name: Nikdan, Mahdi
  id: 66374281-f394-11eb-9cf6-869147deecc0
  last_name: Nikdan
- first_name: Andrei
  full_name: Panferov, Andrei
  id: 2c18daae-4dbe-11ef-8491-98ce2d960f09
  last_name: Panferov
- first_name: Ilia
  full_name: Markov, Ilia
  id: D0CF4148-C985-11E9-8066-0BDEE5697425
  last_name: Markov
- first_name: Nir
  full_name: Shavit, Nir
  last_name: Shavit
- first_name: Dan-Adrian
  full_name: Alistarh, Dan-Adrian
  id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
  last_name: Alistarh
  orcid: 0000-0003-3650-940X
citation:
  ama: 'Nicolicioiu A, Iofinova EB, Jovanovic A, et al. <i>Panza: Investigating the
    Feasibility of Fully-Local Personalized Text Generation</i>. OpenReview; 2026.'
  apa: 'Nicolicioiu, A., Iofinova, E. B., Jovanovic, A., Kurtic, E., Nikdan, M., Panferov,
    A., … Alistarh, D.-A. (2026). <i>Panza: Investigating the feasibility of fully-local
    personalized text generation</i>. <i>Third Conference on Parsimony and Learning
    (Proceedings Track)</i>. Tübíngen, Germany: OpenReview.'
  chicago: 'Nicolicioiu, Armand, Eugenia B Iofinova, Andrej Jovanovic, Eldar Kurtic,
    Mahdi Nikdan, Andrei Panferov, Ilia Markov, Nir Shavit, and Dan-Adrian Alistarh.
    <i>Panza: Investigating the Feasibility of Fully-Local Personalized Text Generation</i>.
    <i>Third Conference on Parsimony and Learning (Proceedings Track)</i>. OpenReview,
    2026.'
  ieee: 'A. Nicolicioiu <i>et al.</i>, <i>Panza: Investigating the feasibility of
    fully-local personalized text generation</i>. OpenReview, 2026.'
  ista: 'Nicolicioiu A, Iofinova EB, Jovanovic A, Kurtic E, Nikdan M, Panferov A,
    Markov I, Shavit N, Alistarh D-A. 2026. Panza: Investigating the feasibility of
    fully-local personalized text generation, OpenReview,p.'
  mla: 'Nicolicioiu, Armand, et al. “Panza: Investigating the Feasibility of Fully-Local
    Personalized Text Generation.” <i>Third Conference on Parsimony and Learning (Proceedings
    Track)</i>, 81, OpenReview, 2026.'
  short: 'A. Nicolicioiu, E.B. Iofinova, A. Jovanovic, E. Kurtic, M. Nikdan, A. Panferov,
    I. Markov, N. Shavit, D.-A. Alistarh, Panza: Investigating the Feasibility of
    Fully-Local Personalized Text Generation, OpenReview, 2026.'
conference:
  end_date: 2026-03-26
  location: Tübíngen, Germany
  name: 'CPAL: Conference on Parsimony and Learning'
  start_date: 2026-03-23
corr_author: '1'
date_created: 2026-05-11T08:50:28Z
date_published: 2026-03-06T00:00:00Z
date_updated: 2026-05-19T11:20:27Z
day: '06'
department:
- _id: GradSch
- _id: DaAl
keyword:
- LLMs
- PEFT
- LoRA
- personalization
- efficient ML
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://openreview.net/pdf?id=soFWnTqd23
month: '03'
oa: 1
oa_version: Accepted Version
publication: Third Conference on Parsimony and Learning (Proceedings Track)
publication_status: published
publisher: OpenReview
quality_controlled: '1'
related_material:
  record:
  - id: '21854'
    relation: dissertation_contains
    status: public
status: public
title: 'Panza: Investigating the feasibility of fully-local personalized text generation'
tmp:
  image: /images/cc_by.png
  legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
  name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
  short: CC BY (4.0)
type: conference_poster
user_id: 8b945eb4-e2f2-11eb-945a-df72226e66a9
year: '2026'
...
---
_id: '18117'
abstract:
- lang: eng
  text: "We investigate parameter-efficient fine-tuning (PEFT) methods that can provide
    good accuracy under limited computational and memory budgets in the context of
    large language models (LLMs). We present a new PEFT method called Robust Adaptation
    (RoSA) inspired by robust principal component analysis that jointly trains low-rank\r\n
    and highly-sparse components on top of a set of fixed pretrained weights to efficiently
    approximate the performance of a full-fine-tuning (FFT) solution. Across a series
    of challenging generative tasks such as grade-school math and SQL query generation,
    which require fine-tuning for good performance, we show that RoSA outperforms
    LoRA, pure sparse fine-tuning, and alternative hybrid methods at the same parameter
    budget, and can even recover the performance of FFT on some tasks. We provide
    system support for RoSA to complement the training algorithm, specifically in
    the form of sparse GPU kernels which enable memory- and computationally-efficient
    training, and show that it is also compatible with low-precision base weights,
    resulting in the first joint representation combining quantization, low-rank and
    sparse approximations. Our code is available at https://github.com/IST-DASLab/RoSA."
acknowledgement: The authors would like to thank Eldar Kurtic for experimental support
  and useful suggestions throughout the project
article_processing_charge: No
arxiv: 1
author:
- first_name: Mahdi
  full_name: Nikdan, Mahdi
  id: 66374281-f394-11eb-9cf6-869147deecc0
  last_name: Nikdan
- first_name: Soroush
  full_name: Tabesh, Soroush
  id: 06000900-6068-11ef-8d61-c2472ef2e752
  last_name: Tabesh
  orcid: 0009-0003-4119-6281
- first_name: Elvir
  full_name: Crncevic, Elvir
  id: 41888001-440d-11ef-8299-d0e838b8185e
  last_name: Crncevic
- first_name: Dan-Adrian
  full_name: Alistarh, Dan-Adrian
  id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
  last_name: Alistarh
  orcid: 0000-0003-3650-940X
citation:
  ama: 'Nikdan M, Tabesh S, Crncevic E, Alistarh D-A. RoSA: Accurate parameter-efficient
    fine-tuning via robust adaptation. In: <i>Proceedings of the 41st International
    Conference on Machine Learning</i>. Vol 235. ML Research Press; 2024:38187-38206.'
  apa: 'Nikdan, M., Tabesh, S., Crncevic, E., &#38; Alistarh, D.-A. (2024). RoSA:
    Accurate parameter-efficient fine-tuning via robust adaptation. In <i>Proceedings
    of the 41st International Conference on Machine Learning</i> (Vol. 235, pp. 38187–38206).
    Vienna, Austria: ML Research Press.'
  chicago: 'Nikdan, Mahdi, Soroush Tabesh, Elvir Crncevic, and Dan-Adrian Alistarh.
    “RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation.” In <i>Proceedings
    of the 41st International Conference on Machine Learning</i>, 235:38187–206. ML
    Research Press, 2024.'
  ieee: 'M. Nikdan, S. Tabesh, E. Crncevic, and D.-A. Alistarh, “RoSA: Accurate parameter-efficient
    fine-tuning via robust adaptation,” in <i>Proceedings of the 41st International
    Conference on Machine Learning</i>, Vienna, Austria, 2024, vol. 235, pp. 38187–38206.'
  ista: 'Nikdan M, Tabesh S, Crncevic E, Alistarh D-A. 2024. RoSA: Accurate parameter-efficient
    fine-tuning via robust adaptation. Proceedings of the 41st International Conference
    on Machine Learning. ICML: International Conference on Machine Learning vol. 235,
    38187–38206.'
  mla: 'Nikdan, Mahdi, et al. “RoSA: Accurate Parameter-Efficient Fine-Tuning via
    Robust Adaptation.” <i>Proceedings of the 41st International Conference on Machine
    Learning</i>, vol. 235, ML Research Press, 2024, pp. 38187–206.'
  short: M. Nikdan, S. Tabesh, E. Crncevic, D.-A. Alistarh, in:, Proceedings of the
    41st International Conference on Machine Learning, ML Research Press, 2024, pp.
    38187–38206.
conference:
  end_date: 2024-07-27
  location: Vienna, Austria
  name: 'ICML: International Conference on Machine Learning'
  start_date: 2024-07-21
corr_author: '1'
date_created: 2024-09-22T22:01:44Z
date_published: 2024-09-01T00:00:00Z
date_updated: 2024-10-01T08:22:01Z
day: '01'
department:
- _id: DaAl
- _id: GradSch
external_id:
  arxiv:
  - '2401.04679'
intvolume: '       235'
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://doi.org/10.48550/arXiv.2401.04679
month: '09'
oa: 1
oa_version: Preprint
page: 38187-38206
publication: Proceedings of the 41st International Conference on Machine Learning
publication_identifier:
  eissn:
  - 2640-3498
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
related_material:
  link:
  - relation: software
    url: https://github.com/IST-DASLab/RoSA
scopus_import: '1'
status: public
title: 'RoSA: Accurate parameter-efficient fine-tuning via robust adaptation'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 235
year: '2024'
...
---
_id: '14460'
abstract:
- lang: eng
  text: We provide an efficient implementation of the backpropagation algorithm, specialized
    to the case where the weights of the neural network being trained are sparse.
    Our algorithm is general, as it applies to arbitrary (unstructured) sparsity and
    common layer types (e.g., convolutional or linear). We provide a fast vectorized
    implementation on commodity CPUs, and show that it can yield speedups in end-to-end
    runtime experiments, both in transfer learning using already-sparsified networks,
    and in training sparse networks from scratch. Thus, our results provide the first
    support for sparse training on commodity hardware.
acknowledgement: 'We would like to thank Elias Frantar for his valuable assistance
  and support at the outset of this project, and the anonymous ICML and SNN reviewers
  for very constructive feedback. EI was supported in part by the FWF DK VGSCO, grant
  agreement number W1260-N35. DA acknowledges generous ERC support, via Starting Grant
  805223 ScaleML. '
alternative_title:
- PMLR
article_processing_charge: No
arxiv: 1
author:
- first_name: Mahdi
  full_name: Nikdan, Mahdi
  id: 66374281-f394-11eb-9cf6-869147deecc0
  last_name: Nikdan
- first_name: Tommaso
  full_name: Pegolotti, Tommaso
  last_name: Pegolotti
- first_name: Eugenia B
  full_name: Iofinova, Eugenia B
  id: f9a17499-f6e0-11ea-865d-fdf9a3f77117
  last_name: Iofinova
  orcid: 0000-0002-7778-3221
- first_name: Eldar
  full_name: Kurtic, Eldar
  id: 47beb3a5-07b5-11eb-9b87-b108ec578218
  last_name: Kurtic
- first_name: Dan-Adrian
  full_name: Alistarh, Dan-Adrian
  id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
  last_name: Alistarh
  orcid: 0000-0003-3650-940X
citation:
  ama: 'Nikdan M, Pegolotti T, Iofinova EB, Kurtic E, Alistarh D-A. SparseProp: Efficient
    sparse backpropagation for faster training of neural networks at the edge. In:
    <i>Proceedings of the 40th International Conference on Machine Learning</i>. Vol
    202. ML Research Press; 2023:26215-26227.'
  apa: 'Nikdan, M., Pegolotti, T., Iofinova, E. B., Kurtic, E., &#38; Alistarh, D.-A.
    (2023). SparseProp: Efficient sparse backpropagation for faster training of neural
    networks at the edge. In <i>Proceedings of the 40th International Conference on
    Machine Learning</i> (Vol. 202, pp. 26215–26227). Honolulu, Hawaii, HI, United
    States: ML Research Press.'
  chicago: 'Nikdan, Mahdi, Tommaso Pegolotti, Eugenia B Iofinova, Eldar Kurtic, and
    Dan-Adrian Alistarh. “SparseProp: Efficient Sparse Backpropagation for Faster
    Training of Neural Networks at the Edge.” In <i>Proceedings of the 40th International
    Conference on Machine Learning</i>, 202:26215–27. ML Research Press, 2023.'
  ieee: 'M. Nikdan, T. Pegolotti, E. B. Iofinova, E. Kurtic, and D.-A. Alistarh, “SparseProp:
    Efficient sparse backpropagation for faster training of neural networks at the
    edge,” in <i>Proceedings of the 40th International Conference on Machine Learning</i>,
    Honolulu, Hawaii, HI, United States, 2023, vol. 202, pp. 26215–26227.'
  ista: 'Nikdan M, Pegolotti T, Iofinova EB, Kurtic E, Alistarh D-A. 2023. SparseProp:
    Efficient sparse backpropagation for faster training of neural networks at the
    edge. Proceedings of the 40th International Conference on Machine Learning. ICML:
    International Conference on Machine Learning, PMLR, vol. 202, 26215–26227.'
  mla: 'Nikdan, Mahdi, et al. “SparseProp: Efficient Sparse Backpropagation for Faster
    Training of Neural Networks at the Edge.” <i>Proceedings of the 40th International
    Conference on Machine Learning</i>, vol. 202, ML Research Press, 2023, pp. 26215–27.'
  short: M. Nikdan, T. Pegolotti, E.B. Iofinova, E. Kurtic, D.-A. Alistarh, in:, Proceedings
    of the 40th International Conference on Machine Learning, ML Research Press, 2023,
    pp. 26215–26227.
conference:
  end_date: 2023-07-29
  location: Honolulu, Hawaii, HI, United States
  name: 'ICML: International Conference on Machine Learning'
  start_date: 2023-07-23
corr_author: '1'
date_created: 2023-10-29T23:01:17Z
date_published: 2023-07-30T00:00:00Z
date_updated: 2025-04-14T07:49:12Z
day: '30'
department:
- _id: DaAl
ec_funded: 1
external_id:
  arxiv:
  - '2302.04852'
intvolume: '       202'
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://doi.org/10.48550/arXiv.2302.04852
month: '07'
oa: 1
oa_version: Preprint
page: 26215-26227
project:
- _id: 268A44D6-B435-11E9-9278-68D0E5697425
  call_identifier: H2020
  grant_number: '805223'
  name: Elastic Coordination for Scalable Machine Learning
publication: Proceedings of the 40th International Conference on Machine Learning
publication_identifier:
  eissn:
  - 2640-3498
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'SparseProp: Efficient sparse backpropagation for faster training of neural
  networks at the edge'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 202
year: '2023'
...