---
OA_place: publisher
OA_type: gold
_id: '22146'
abstract:
- lang: eng
text: We study differentially private model training with stochastic gradient descent
under learning rate scheduling and correlated noise. Although correlated noise,
in particular via matrix factorizations, has been shown to improve accuracy, prior
theoretical work focused primarily on the prefix-sum workload. That workload assumes
a constant learning rate, whereas in practice learning rate schedules are widely
used to accelerate training and improve convergence. We close this gap by deriving
general upper and lower bounds for a broad class of learning rate schedules in
both single- and multi-epoch settings. Building on these results, we propose a
learning-rate-aware factorization that achieves improvements over prefix-sum factorizations
under both MaxSE and MeanSE error metrics. Our theoretical analysis yields memory-efficient
constructions suitable for practical deployment, and experiments on CIFAR-10 and
IMDB datasets confirm that schedule-aware factorizations improve accuracy in private
training.
acknowledgement: "We thank Rasmus Pagh, Christoph Lampert and Jalaj Upadhyay for valuable\r\ncomments
on an early draft. We thank Ryan Mckenna for a fruitful discussion on the experiment\r\ndesign.
We thank Antti Honkela for sharing insights on learning rate scheduling and DP.\r\nNikita
P. Kalinin: Funded in part by the Austrian Science Fund (FWF) [10.55776/COE12].\r\nJoel
Daniel Andersson: Funded by the European Union. Views and opinions expressed are
however\r\nthose of the author(s) only and do not necessarily reflect those of the
European Union or the European\r\nResearch Council Executive Agency. Neither the
European Union nor the granting authority can be\r\nheld responsible for them. This
project has received funding from the European Research Council\r\n(ERC) under the
European Union’s Horizon 2020 research and innovation programme (MoDynStruct,\r\nNo.
101019564). Additional funding by Providentia, a Data Science Distinguished Investigator
grant\r\nfrom Novo Nordisk Fonden, with additional support from VILLUM Investigator
grant 54451.\r\n"
alternative_title:
- LIPIcs
article_number: 2:1-2:21
article_processing_charge: No
arxiv: 1
author:
- first_name: Nikita
full_name: Kalinin, Nikita
id: 4b14526e-14d2-11ed-ba64-c14c9553d137
last_name: Kalinin
- first_name: Joel D
full_name: Andersson, Joel D
id: 4a893819-d954-11f0-89b1-e360bad9ccc5
last_name: Andersson
citation:
ama: 'Kalinin N, Andersson JD. Learning rate scheduling with matrix factorization
for private training. In: 7th Symposium on Foundations of Responsible Computing.
Vol 368. Schloss Dagstuhl - Leibniz-Zentrum für Informatik; 2026. doi:10.4230/LIPIcs.FORC.2026.2'
apa: 'Kalinin, N., & Andersson, J. D. (2026). Learning rate scheduling with
matrix factorization for private training. In 7th Symposium on Foundations
of Responsible Computing (Vol. 368). Cambridge, MA; United States: Schloss
Dagstuhl - Leibniz-Zentrum für Informatik. https://doi.org/10.4230/LIPIcs.FORC.2026.2'
chicago: Kalinin, Nikita, and Joel D Andersson. “Learning Rate Scheduling with Matrix
Factorization for Private Training.” In 7th Symposium on Foundations of Responsible
Computing, Vol. 368. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2026.
https://doi.org/10.4230/LIPIcs.FORC.2026.2.
ieee: N. Kalinin and J. D. Andersson, “Learning rate scheduling with matrix factorization
for private training,” in 7th Symposium on Foundations of Responsible Computing,
Cambridge, MA; United States, 2026, vol. 368.
ista: 'Kalinin N, Andersson JD. 2026. Learning rate scheduling with matrix factorization
for private training. 7th Symposium on Foundations of Responsible Computing. FORC:
Symposium on Foundations of Responsible Computing, LIPIcs, vol. 368, 2:1-2:21.'
mla: Kalinin, Nikita, and Joel D. Andersson. “Learning Rate Scheduling with Matrix
Factorization for Private Training.” 7th Symposium on Foundations of Responsible
Computing, vol. 368, 2:1-2:21, Schloss Dagstuhl - Leibniz-Zentrum für Informatik,
2026, doi:10.4230/LIPIcs.FORC.2026.2.
short: N. Kalinin, J.D. Andersson, in:, 7th Symposium on Foundations of Responsible
Computing, Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2026.
conference:
end_date: 2026-06-05
location: Cambridge, MA; United States
name: 'FORC: Symposium on Foundations of Responsible Computing'
start_date: 2026-06-03
corr_author: '1'
das_tickbox: '0'
date_created: 2026-06-28T22:01:34Z
date_published: 2026-06-01T00:00:00Z
date_updated: 2026-06-29T06:56:34Z
day: '01'
ddc:
- '000'
department:
- _id: ChLa
- _id: GradSch
- _id: MoHe
doi: 10.4230/LIPIcs.FORC.2026.2
ec_funded: 1
external_id:
arxiv:
- '2511.17994'
file:
- access_level: open_access
checksum: c661f016d3861a1c1b590b87a744d087
content_type: application/pdf
creator: dernst
date_created: 2026-06-29T06:55:23Z
date_updated: 2026-06-29T06:55:23Z
file_id: '22149'
file_name: 2026_LIPIcsFORC_Kalinin.pdf
file_size: 1231914
relation: main_file
success: 1
file_date_updated: 2026-06-29T06:55:23Z
has_accepted_license: '1'
intvolume: ' 368'
keyword:
- differential privacy
- machine learning
- matrix factorization
language:
- iso: eng
month: '06'
oa: 1
oa_version: Published Version
project:
- _id: bd9ca328-d553-11ed-ba76-dc4f890cfe62
call_identifier: H2020
grant_number: '101019564'
name: The design and evaluation of modern fully dynamic data structures
publication: 7th Symposium on Foundations of Responsible Computing
publication_identifier:
eissn:
- 1868-8969
isbn:
- '9783959774192'
publication_status: published
publisher: Schloss Dagstuhl - Leibniz-Zentrum für Informatik
quality_controlled: '1'
researchdata_availability: no
scopus_import: '1'
status: public
supplementarymaterial: no
title: Learning rate scheduling with matrix factorization for private training
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 368
year: '2026'
...