---
OA_place: publisher
OA_type: free access
_id: '19713'
abstract:
- lang: eng
  text: Distributed optimization is the standard way of speeding up machine learning
    training, and most of the research in the area focuses on distributed first-order,
    gradient-based methods. Yet, there are settings where some computationally-bounded
    nodes may not be able to implement first-order, gradient-based optimization, while
    they could still contribute to joint optimization tasks. In this paper, we initiate
    the study of hybrid decentralized optimization, studying settings where nodes
    with zeroth-order and first-order optimization capabilities co-exist in a distributed
    system, and attempt to jointly solve an optimization task over some data distribution.
    We essentially show that, under reasonable parameter settings, such a system can
    not only withstand noisier zeroth-order agents but can even benefit from integrating
    such agents into the optimization process, rather than ignoring their information.
    At the core of our approach is a new analysis of distributed optimization with
    noisy and possibly-biased gradient estimators, which may be of independent interest.
    Our results hold for both convex and non-convex objectives. Experimental results
    on standard optimization tasks confirm our analysis, showing that hybrid first-zeroth
    order optimization can be practical, even when training deep neural networks.
acknowledgement: "This project has received funding from the European Research Council
  (ERC) under the European Union’s Horizon 2020 research and innovation programme
  (grant agreement\r\nNo 805223 ScaleML). The authors would like to acknowledge Eugenia
  Iofinova for useful discussions during the inception of this project."
article_processing_charge: No
article_type: original
arxiv: 1
author:
- first_name: Shayan
  full_name: Talaei, Shayan
  last_name: Talaei
- first_name: Matin
  full_name: Ansaripour, Matin
  last_name: Ansaripour
- first_name: Giorgi
  full_name: Nadiradze, Giorgi
  id: 3279A00C-F248-11E8-B48F-1D18A9856A87
  last_name: Nadiradze
  orcid: 0000-0001-5634-0731
- first_name: Dan-Adrian
  full_name: Alistarh, Dan-Adrian
  id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
  last_name: Alistarh
  orcid: 0000-0003-3650-940X
citation:
  ama: 'Talaei S, Ansaripour M, Nadiradze G, Alistarh D-A. Hybrid decentralized optimization:
    Leveraging both first- and zeroth-order optimizers for faster convergence. <i>Proceedings
    of the 39th AAAI Conference on Artificial Intelligence</i>. 2025;39(19):20778-20786.
    doi:<a href="https://doi.org/10.1609/aaai.v39i19.34290">10.1609/aaai.v39i19.34290</a>'
  apa: 'Talaei, S., Ansaripour, M., Nadiradze, G., &#38; Alistarh, D.-A. (2025). Hybrid
    decentralized optimization: Leveraging both first- and zeroth-order optimizers
    for faster convergence. <i>Proceedings of the 39th AAAI Conference on Artificial
    Intelligence</i>. Association for the Advancement of Artificial Intelligence.
    <a href="https://doi.org/10.1609/aaai.v39i19.34290">https://doi.org/10.1609/aaai.v39i19.34290</a>'
  chicago: 'Talaei, Shayan, Matin Ansaripour, Giorgi Nadiradze, and Dan-Adrian Alistarh.
    “Hybrid Decentralized Optimization: Leveraging Both First- and Zeroth-Order Optimizers
    for Faster Convergence.” <i>Proceedings of the 39th AAAI Conference on Artificial
    Intelligence</i>. Association for the Advancement of Artificial Intelligence,
    2025. <a href="https://doi.org/10.1609/aaai.v39i19.34290">https://doi.org/10.1609/aaai.v39i19.34290</a>.'
  ieee: 'S. Talaei, M. Ansaripour, G. Nadiradze, and D.-A. Alistarh, “Hybrid decentralized
    optimization: Leveraging both first- and zeroth-order optimizers for faster convergence,”
    <i>Proceedings of the 39th AAAI Conference on Artificial Intelligence</i>, vol.
    39, no. 19. Association for the Advancement of Artificial Intelligence, pp. 20778–20786,
    2025.'
  ista: 'Talaei S, Ansaripour M, Nadiradze G, Alistarh D-A. 2025. Hybrid decentralized
    optimization: Leveraging both first- and zeroth-order optimizers for faster convergence.
    Proceedings of the 39th AAAI Conference on Artificial Intelligence. 39(19), 20778–20786.'
  mla: 'Talaei, Shayan, et al. “Hybrid Decentralized Optimization: Leveraging Both
    First- and Zeroth-Order Optimizers for Faster Convergence.” <i>Proceedings of
    the 39th AAAI Conference on Artificial Intelligence</i>, vol. 39, no. 19, Association
    for the Advancement of Artificial Intelligence, 2025, pp. 20778–86, doi:<a href="https://doi.org/10.1609/aaai.v39i19.34290">10.1609/aaai.v39i19.34290</a>.'
  short: S. Talaei, M. Ansaripour, G. Nadiradze, D.-A. Alistarh, Proceedings of the
    39th AAAI Conference on Artificial Intelligence 39 (2025) 20778–20786.
corr_author: '1'
date_created: 2025-05-19T14:15:35Z
date_published: 2025-04-11T00:00:00Z
date_updated: 2026-02-16T12:34:44Z
day: '11'
department:
- _id: DaAl
doi: 10.1609/aaai.v39i19.34290
ec_funded: 1
external_id:
  arxiv:
  - '2210.07703'
intvolume: '        39'
issue: '19'
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://doi.org/10.1609/aaai.v39i19.34290
month: '04'
oa: 1
oa_version: Preprint
page: 20778-20786
project:
- _id: 268A44D6-B435-11E9-9278-68D0E5697425
  call_identifier: H2020
  grant_number: '805223'
  name: Elastic Coordination for Scalable Machine Learning
publication: Proceedings of the 39th AAAI Conference on Artificial Intelligence
publication_identifier:
  eissn:
  - 2374-3468
  issn:
  - 2159-5399
publication_status: published
publisher: Association for the Advancement of Artificial Intelligence
quality_controlled: '1'
related_material:
  link:
  - relation: software
    url: https://github.com/ShayanTalaei/HDO
scopus_import: '1'
status: public
title: 'Hybrid decentralized optimization: Leveraging both first- and zeroth-order
  optimizers for faster convergence'
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 39
year: '2025'
...