Hybrid decentralized optimization: Leveraging both first- and zeroth-order optimizers for faster convergence Talaei, Shayan Ansaripour, Matin Nadiradze, Giorgi ; https://orcid.org/0000-0001-5634-0731 Alistarh, Dan-Adrian ; https://orcid.org/0000-0003-3650-940X Distributed optimization is the standard way of speeding up machine learning training, and most of the research in the area focuses on distributed first-order, gradient-based methods. Yet, there are settings where some computationally-bounded nodes may not be able to implement first-order, gradient-based optimization, while they could still contribute to joint optimization tasks. In this paper, we initiate the study of hybrid decentralized optimization, studying settings where nodes with zeroth-order and first-order optimization capabilities co-exist in a distributed system, and attempt to jointly solve an optimization task over some data distribution. We essentially show that, under reasonable parameter settings, such a system can not only withstand noisier zeroth-order agents but can even benefit from integrating such agents into the optimization process, rather than ignoring their information. At the core of our approach is a new analysis of distributed optimization with noisy and possibly-biased gradient estimators, which may be of independent interest. Our results hold for both convex and non-convex objectives. Experimental results on standard optimization tasks confirm our analysis, showing that hybrid first-zeroth order optimization can be practical, even when training deep neural networks. Association for the Advancement of Artificial Intelligence 2025 info:eu-repo/semantics/article doc-type:Article Article http://purl.org/coar/resource_type/c_2df8fbb1 https://research-explorer.ista.ac.at/record/19713 Talaei S, Ansaripour M, Nadiradze G, Alistarh D-A. Hybrid decentralized optimization: Leveraging both first- and zeroth-order optimizers for faster convergence. <i>Proceedings of the 39th AAAI Conference on Artificial Intelligence</i>. 2025;39(19):20778-20786. doi:<a href="https://doi.org/10.1609/aaai.v39i19.34290">10.1609/aaai.v39i19.34290</a> eng info:eu-repo/semantics/altIdentifier/doi/10.1609/aaai.v39i19.34290 info:eu-repo/semantics/altIdentifier/issn/2159-5399 info:eu-repo/semantics/altIdentifier/e-issn/2374-3468 info:eu-repo/semantics/altIdentifier/arxiv/2210.07703 info:eu-repo/semantics/openAccess