---
res:
  bibo_abstract:
  - Quantized training of Large Language Models (LLMs) remains an open challenge,
    as maintaining accuracy while performing all matrix multiplications in low precision
    has proven difficult. This is particularly the case when fine-tuning pre-trained
    models, which can have large weight, activation, and error (output gradient) outlier
    values that make lower-precision optimization difficult. To address this, we present
    HALO, a new quantization-aware training approach for Transformers that enables
    accurate and efficient low-precision training by combining 1) strategic placement
    of Hadamard rotations in both forward and backward passes, which mitigate outliers,
    2) high-performance kernel support, and 3) FSDP integration for low-precision
    communication. Our approach ensures that all large matrix multiplications during
    the forward and backward passes are executed in lower precision. Applied to LLaMa
    models, HALO achieves near-full-precision-equivalent results during fine-tuning
    on various tasks, while delivering up to 1.41x end-to-end speedup for full fine-tuning
    on RTX 4090 GPUs. HALO efficiently supports both standard and parameter-efficient
    fine-tuning (PEFT). Our results demonstrate the first practical approach to fully
    quantized LLM fine-tuning that maintains accuracy in INT8 and FP6 precision, while
    delivering performance benefits.@eng
  bibo_authorlist:
  - foaf_Person:
      foaf_givenName: Saleh
      foaf_name: Ashkboos, Saleh
      foaf_surname: Ashkboos
  - foaf_Person:
      foaf_givenName: Mahdi
      foaf_name: Nikdan, Mahdi
      foaf_surname: Nikdan
      foaf_workInfoHomepage: http://www.librecat.org/personId=66374281-f394-11eb-9cf6-869147deecc0
  - foaf_Person:
      foaf_givenName: Soroush
      foaf_name: Tabesh, Soroush
      foaf_surname: Tabesh
      foaf_workInfoHomepage: http://www.librecat.org/personId=06000900-6068-11ef-8d61-c2472ef2e752
    orcid: 0009-0003-4119-6281
  - foaf_Person:
      foaf_givenName: Roberto
      foaf_name: Lopez Castro, Roberto
      foaf_surname: Lopez Castro
      foaf_workInfoHomepage: http://www.librecat.org/personId=18495c31-fb57-11ef-ba0d-c290e10e394e
  - foaf_Person:
      foaf_givenName: Torsten
      foaf_name: Hoefler, Torsten
      foaf_surname: Hoefler
  - foaf_Person:
      foaf_givenName: Dan-Adrian
      foaf_name: Alistarh, Dan-Adrian
      foaf_surname: Alistarh
      foaf_workInfoHomepage: http://www.librecat.org/personId=4A899BFC-F248-11E8-B48F-1D18A9856A87
    orcid: 0000-0003-3650-940X
  bibo_doi: 10.52202/085713-3966
  bibo_volume: 38
  dct_date: 2025^xs_gYear
  dct_isPartOf:
  - http://id.crossref.org/issn/1049-5258
  - http://id.crossref.org/issn/9798331338275
  dct_language: eng
  dct_publisher: Neural Information Processing Systems Foundation@
  dct_title: 'HALO: Hadamard-assisted lower-precision optimization for LLMs@'
...