---
OA_place: publisher
OA_type: diamond
_id: '20034'
abstract:
- lang: eng
  text: We introduce LDAdam, a memory-efficient optimizer for training large models,
    that performs adaptive optimization steps within lower dimensional subspaces,
    while consistently exploring the full parameter space during training. This strategy
    keeps the optimizer's memory footprint to a fraction of the model size. LDAdam
    relies on a new projection-aware update rule for the optimizer states that allows
    for transitioning between subspaces, i.e., estimation of the statistics of the
    projected gradients. To mitigate the errors due to low-rank projection, LDAdam
    integrates a new generalized error feedback mechanism, which explicitly accounts
    for both gradient and optimizer state compression. We prove the convergence of
    LDAdam under standard assumptions, and provide empirical evidence that LDAdam
    allows for efficient fine-tuning and pre-training of language models.
article_processing_charge: No
arxiv: 1
author:
- first_name: Thomas
  full_name: Robert, Thomas
  last_name: Robert
- first_name: Mher
  full_name: Safaryan, Mher
  id: dd546b39-0804-11ed-9c55-ef075c39778d
  last_name: Safaryan
- first_name: Ionut-Vlad
  full_name: Modoranu, Ionut-Vlad
  id: 449f7a18-f128-11eb-9611-9b430c0c6333
  last_name: Modoranu
- first_name: Dan-Adrian
  full_name: Alistarh, Dan-Adrian
  id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
  last_name: Alistarh
  orcid: 0000-0003-3650-940X
citation:
  ama: 'Robert T, Safaryan M, Modoranu I-V, Alistarh D-A. LDAdam: Adaptive optimization
    from low-dimensional gradient statistics. In: <i>13th International Conference
    on Learning Representations</i>. ICLR; 2025:101877-101913.'
  apa: 'Robert, T., Safaryan, M., Modoranu, I.-V., &#38; Alistarh, D.-A. (2025). LDAdam:
    Adaptive optimization from low-dimensional gradient statistics. In <i>13th International
    Conference on Learning Representations</i> (pp. 101877–101913). Singapore, Singapore:
    ICLR.'
  chicago: 'Robert, Thomas, Mher Safaryan, Ionut-Vlad Modoranu, and Dan-Adrian Alistarh.
    “LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics.” In <i>13th
    International Conference on Learning Representations</i>, 101877–913. ICLR, 2025.'
  ieee: 'T. Robert, M. Safaryan, I.-V. Modoranu, and D.-A. Alistarh, “LDAdam: Adaptive
    optimization from low-dimensional gradient statistics,” in <i>13th International
    Conference on Learning Representations</i>, Singapore, Singapore, 2025, pp. 101877–101913.'
  ista: 'Robert T, Safaryan M, Modoranu I-V, Alistarh D-A. 2025. LDAdam: Adaptive
    optimization from low-dimensional gradient statistics. 13th International Conference
    on Learning Representations. ICLR: International Conference on Learning Representations,
    101877–101913.'
  mla: 'Robert, Thomas, et al. “LDAdam: Adaptive Optimization from Low-Dimensional
    Gradient Statistics.” <i>13th International Conference on Learning Representations</i>,
    ICLR, 2025, pp. 101877–913.'
  short: T. Robert, M. Safaryan, I.-V. Modoranu, D.-A. Alistarh, in:, 13th International
    Conference on Learning Representations, ICLR, 2025, pp. 101877–101913.
conference:
  end_date: 2025-04-28
  location: Singapore, Singapore
  name: 'ICLR: International Conference on Learning Representations'
  start_date: 2025-04-24
corr_author: '1'
date_created: 2025-07-20T22:02:02Z
date_published: 2025-04-01T00:00:00Z
date_updated: 2025-08-04T08:41:10Z
day: '01'
ddc:
- '000'
department:
- _id: DaAl
external_id:
  arxiv:
  - '2410.16103'
file:
- access_level: open_access
  checksum: 9327d82569358d7bf1c3ec1a9952e721
  content_type: application/pdf
  creator: dernst
  date_created: 2025-08-04T08:39:51Z
  date_updated: 2025-08-04T08:39:51Z
  file_id: '20113'
  file_name: 2025_ICLR_Robert.pdf
  file_size: 1346111
  relation: main_file
  success: 1
file_date_updated: 2025-08-04T08:39:51Z
has_accepted_license: '1'
language:
- iso: eng
month: '04'
oa: 1
oa_version: Published Version
page: 101877-101913
publication: 13th International Conference on Learning Representations
publication_identifier:
  isbn:
  - '9798331320850'
publication_status: published
publisher: ICLR
quality_controlled: '1'
related_material:
  link:
  - relation: software
    url: https://github.com/IST-DASLab/LDAdam
scopus_import: '1'
status: public
title: 'LDAdam: Adaptive optimization from low-dimensional gradient statistics'
tmp:
  image: /images/cc_by.png
  legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
  name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
  short: CC BY (4.0)
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2025'
...