[{"_id":"21324","file":[{"access_level":"open_access","creator":"dernst","date_created":"2026-02-19T08:04:38Z","relation":"main_file","file_id":"21335","date_updated":"2026-02-19T08:04:38Z","file_name":"2025_ICML_Bombari.pdf","file_size":887526,"success":1,"content_type":"application/pdf","checksum":"d4ba4f7717b362ca38878f45e57bd643"}],"date_published":"2025-07-30T00:00:00Z","publisher":"ML Research Press","acknowledgement":"Marco Mondelli is funded by the European Union (ERC, INF2, project number 101161364). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. Simone Bombari is supported by a Google PhD fellowship. The authors would like to thank GuanWen Qiu for helpful discussions.","oa_version":"Published Version","conference":{"start_date":"2025-07-13","name":"ICML: International Conference on Machine Learning","location":"Vancouver, Canada","end_date":"2025-07-19"},"title":"Spurious correlations in high dimensional regression: The roles of regularization, simplicity bias and over-parameterization","department":[{"_id":"MaMo"}],"project":[{"_id":"911e6d1f-16d5-11f0-9cad-c5c68c6a1cdf","grant_number":"101161364","name":"Inference in High Dimensions: Light-speed Algorithms and Information Limits"},{"_id":"92099302-16d5-11f0-9cad-f9a785f54fbd","name":"Trustworthy Deep Learning Theory: Private Over-Parameterized Models and Robust LLMs"}],"date_updated":"2026-02-19T08:08:55Z","OA_place":"publisher","alternative_title":["PMLR"],"user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","citation":{"mla":"Bombari, Simone, and Marco Mondelli. “Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and over-Parameterization.” <i>Proceedings of the 42nd International Conference on Machine Learning</i>, vol. 267, ML Research Press, 2025, pp. 4839–73.","chicago":"Bombari, Simone, and Marco Mondelli. “Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and over-Parameterization.” In <i>Proceedings of the 42nd International Conference on Machine Learning</i>, 267:4839–73. ML Research Press, 2025.","ieee":"S. Bombari and M. Mondelli, “Spurious correlations in high dimensional regression: The roles of regularization, simplicity bias and over-parameterization,” in <i>Proceedings of the 42nd International Conference on Machine Learning</i>, Vancouver, Canada, 2025, vol. 267, pp. 4839–4873.","ama":"Bombari S, Mondelli M. Spurious correlations in high dimensional regression: The roles of regularization, simplicity bias and over-parameterization. In: <i>Proceedings of the 42nd International Conference on Machine Learning</i>. Vol 267. ML Research Press; 2025:4839-4873.","ista":"Bombari S, Mondelli M. 2025. Spurious correlations in high dimensional regression: The roles of regularization, simplicity bias and over-parameterization. Proceedings of the 42nd International Conference on Machine Learning. ICML: International Conference on Machine Learning, PMLR, vol. 267, 4839–4873.","apa":"Bombari, S., &#38; Mondelli, M. (2025). Spurious correlations in high dimensional regression: The roles of regularization, simplicity bias and over-parameterization. In <i>Proceedings of the 42nd International Conference on Machine Learning</i> (Vol. 267, pp. 4839–4873). Vancouver, Canada: ML Research Press.","short":"S. Bombari, M. Mondelli, in:, Proceedings of the 42nd International Conference on Machine Learning, ML Research Press, 2025, pp. 4839–4873."},"oa":1,"date_created":"2026-02-18T11:58:00Z","arxiv":1,"quality_controlled":"1","file_date_updated":"2026-02-19T08:04:38Z","OA_type":"gold","year":"2025","language":[{"iso":"eng"}],"ddc":["000"],"page":"4839-4873","publication":"Proceedings of the 42nd International Conference on Machine Learning","external_id":{"arxiv":["2502.01347"]},"article_processing_charge":"No","abstract":[{"text":"Learning models have been shown to rely on spurious correlations between non-predictive features and the associated labels in the training data, with negative implications on robustness, bias and fairness. In this work, we provide a statistical characterization of this phenomenon for high-dimensional regression, when the data contains a predictive core feature x and a spurious feature y. Specifically, we quantify the amount of spurious correlations C learned via linear regression, in terms of the data covariance and the strength λ of the ridge regularization. As a consequence, we first capture the simplicity of y through the spectrum of its covariance, and its correlation with x through the Schur complement of the full data covariance. Next, we prove a trade-off between C and the in-distribution test loss L, by showing that the value of λ that minimizes L lies in an interval where C is increasing. Finally, we investigate the effects of over-parameterization via the random features model, by showing its equivalence to regularized linear regression. Our theoretical results are supported by numerical experiments on Gaussian, Color-MNIST, and CIFAR-10 datasets.","lang":"eng"}],"license":"https://creativecommons.org/licenses/by/4.0/","status":"public","author":[{"last_name":"Bombari","full_name":"Bombari, Simone","first_name":"Simone","id":"ca726dda-de17-11ea-bc14-f9da834f63aa"},{"first_name":"Marco","orcid":"0000-0002-3242-7020","id":"27EB676C-8706-11E9-9510-7717E6697425","full_name":"Mondelli, Marco","last_name":"Mondelli"}],"volume":267,"intvolume":"       267","corr_author":"1","day":"30","tmp":{"legal_code_url":"https://creativecommons.org/licenses/by/4.0/legalcode","name":"Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)","short":"CC BY (4.0)","image":"/images/cc_by.png"},"publication_identifier":{"eissn":["2640-3498"]},"type":"conference","publication_status":"published","has_accepted_license":"1","month":"07"},{"_id":"19627","APC_amount":"2754,32 EUR","title":"Privacy for free in the overparameterized regime","acknowledgement":"This research was funded in whole, or in part, by the Austrian Science Fund (FWF) Grant number COE 12. For the purpose of open access, the author has applied a CC BY public copyright license to any Author Accepted Manuscript version arising from this submission. The authors were also supported by the 2019 Lopez-Loreta prize, and Simone Bombari was supported by a Google PhD fellowship. We thank Diyuan Wu, Edwige Cyffers, Francesco Pedrotti, Inbar Seroussi, Nikita P. Kalinin, Pietro Pelliconi, Roodabeh Safavi, Yizhe Zhu, and Zhichao Wang for helpful discussions.","oa_version":"Published Version","date_published":"2025-04-15T00:00:00Z","file":[{"creator":"dernst","access_level":"open_access","date_created":"2025-05-05T07:27:54Z","relation":"main_file","file_id":"19648","date_updated":"2025-05-05T07:27:54Z","file_name":"2025_PNAS_Bombari.pdf","file_size":2328320,"success":1,"checksum":"1ac6f78e368d35a0cafb4d2d9bd63443","content_type":"application/pdf"}],"publisher":"National Academy of Sciences","scopus_import":"1","OA_place":"publisher","date_updated":"2026-05-20T08:23:19Z","citation":{"ista":"Bombari S, Mondelli M. 2025. Privacy for free in the overparameterized regime. Proceedings of the National Academy of Sciences. 122(15), e2423072122.","ama":"Bombari S, Mondelli M. Privacy for free in the overparameterized regime. <i>Proceedings of the National Academy of Sciences</i>. 2025;122(15). doi:<a href=\"https://doi.org/10.1073/pnas.2423072122\">10.1073/pnas.2423072122</a>","apa":"Bombari, S., &#38; Mondelli, M. (2025). Privacy for free in the overparameterized regime. <i>Proceedings of the National Academy of Sciences</i>. National Academy of Sciences. <a href=\"https://doi.org/10.1073/pnas.2423072122\">https://doi.org/10.1073/pnas.2423072122</a>","short":"S. Bombari, M. Mondelli, Proceedings of the National Academy of Sciences 122 (2025).","mla":"Bombari, Simone, and Marco Mondelli. “Privacy for Free in the Overparameterized Regime.” <i>Proceedings of the National Academy of Sciences</i>, vol. 122, no. 15, e2423072122, National Academy of Sciences, 2025, doi:<a href=\"https://doi.org/10.1073/pnas.2423072122\">10.1073/pnas.2423072122</a>.","chicago":"Bombari, Simone, and Marco Mondelli. “Privacy for Free in the Overparameterized Regime.” <i>Proceedings of the National Academy of Sciences</i>. National Academy of Sciences, 2025. <a href=\"https://doi.org/10.1073/pnas.2423072122\">https://doi.org/10.1073/pnas.2423072122</a>.","ieee":"S. Bombari and M. Mondelli, “Privacy for free in the overparameterized regime,” <i>Proceedings of the National Academy of Sciences</i>, vol. 122, no. 15. National Academy of Sciences, 2025."},"oa":1,"user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","project":[{"name":"Prix Lopez-Loretta 2019 - Marco Mondelli","_id":"059876FA-7A3F-11EA-A408-12923DDC885E"},{"name":"Trustworthy Deep Learning Theory: Private Over-Parameterized Models and Robust LLMs","_id":"92099302-16d5-11f0-9cad-f9a785f54fbd"}],"department":[{"_id":"MaMo"}],"file_date_updated":"2025-05-05T07:27:54Z","OA_type":"hybrid","quality_controlled":"1","pmid":1,"issue":"15","arxiv":1,"date_created":"2025-04-27T22:02:13Z","abstract":[{"lang":"eng","text":"Differentially private gradient descent (DP-GD) is a popular algorithm to train deep learning models with provable guarantees on the privacy of the training data. In the last decade, the problem of understanding its performance cost with respect to standard GD has received remarkable attention from the research community, which formally derived upper bounds on the excess population risk  RP  in different learning settings. However, existing bounds typically degrade with over-parameterization, i.e., as the number of parameters  p  gets larger than the number of training samples  n  -- a regime which is ubiquitous in current deep-learning practice. As a result, the lack of theoretical insights leaves practitioners without clear guidance, leading some to reduce the effective number of trainable parameters to improve performance, while others use larger models to achieve better results through scale. In this work, we show that in the popular random features model with quadratic loss, for any sufficiently large  p , privacy can be obtained for free, i.e.,  |RP|=o(1) , not only when the privacy parameter  ε  has constant order, but also in the strongly private setting  ε=o(1) . This challenges the common wisdom that over-parameterization inherently hinders performance in private learning."}],"article_number":"e2423072122","publication":"Proceedings of the National Academy of Sciences","external_id":{"pmid":["40215275"],"isi":["001471214000001"],"arxiv":["2410.14787"]},"article_processing_charge":"Yes (in subscription journal)","language":[{"iso":"eng"}],"ddc":["000"],"year":"2025","status":"public","tmp":{"legal_code_url":"https://creativecommons.org/licenses/by/4.0/legalcode","name":"Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)","short":"CC BY (4.0)","image":"/images/cc_by.png"},"publication_identifier":{"issn":["0027-8424"],"eissn":["1091-6490"]},"corr_author":"1","day":"15","volume":122,"author":[{"id":"ca726dda-de17-11ea-bc14-f9da834f63aa","first_name":"Simone","full_name":"Bombari, Simone","last_name":"Bombari"},{"full_name":"Mondelli, Marco","last_name":"Mondelli","orcid":"0000-0002-3242-7020","first_name":"Marco","id":"27EB676C-8706-11E9-9510-7717E6697425"}],"intvolume":"       122","doi":"10.1073/pnas.2423072122","month":"04","type":"journal_article","publication_status":"published","has_accepted_license":"1","isi":1,"article_type":"original"}]