---
OA_place: publisher
OA_type: gold
_id: '22894'
abstract:
- lang: eng
text: "Large-scale deep learning models are known to memorize parts of the training\r\nset.
In machine learning theory, memorization is often framed as interpolation or\r\nlabel
fitting, and classical results show that this can be achieved when the number\r\nof
parameters p in the model is larger than the number of training samples n. In\r\nthis
work, we consider memorization from the perspective of data reconstruction,\r\ndemonstrating
that this can be achieved when p is larger than dn, where d is\r\nthe dimensionality
of the data. More specifically, we show that, in the random\r\nfeatures model,
when p ≫ dn, the subspace spanned by the training samples in\r\nfeature space
gives sufficient information to identify the individual samples in input\r\nspace.
Our analysis suggests an optimization method to reconstruct the dataset\r\nfrom
the model parameters, and we demonstrate that this method performs well on\r\nvarious
architectures (random features, two-layer fully-connected and deep residual\r\nnetworks).
Our results reveal a law of data reconstruction, according to which the\r\nentire
training dataset can be recovered as p exceeds the threshold dn.\r\n"
acknowledgement: "M.M. is funded by the European Union (ERC, INF2\r\n, project number
101161364). S.B. was supported\r\nby a Google PhD fellowship. L.I. acknowledges
the grant received from the European Union NextGenerationEU (Piano Nazionale di
Ripresa E Resilienza (PNRR)) DM 351 on Trustworthy AI. T.T. &\r\nL.I. acknowledge
the EU project ELSA - European Lighthouse on Secure and Safe AI. This study was\r\ncarried
out within the FAIR - Future Artificial Intelligence Research and received funding
from the\r\nEuropean Union Next-GenerationEU (PIANO NAZIONALE DI RIPRESA E RESILIENZA
(PNRR)\r\n– MISSIONE 4 COMPONENTE 2, INVESTIMENTO 1.3 – D.D. 1555 11/10/2022, PE00000013).\r\nThis
manuscript reflects only the authors’ views and opinions, neither the European Union
nor the\r\nEuropean Commission can be considered responsible for them. The authors
would like to thank\r\nYizhe Zhu for helpful discussions."
article_processing_charge: No
arxiv: 1
author:
- first_name: Leonardo
full_name: Iurada, Leonardo
last_name: Iurada
- first_name: Simone
full_name: Bombari, Simone
id: ca726dda-de17-11ea-bc14-f9da834f63aa
last_name: Bombari
- first_name: Tatiana
full_name: Tommasi, Tatiana
last_name: Tommasi
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: 'Iurada L, Bombari S, Tommasi T, Mondelli M. A law of data reconstruction for
random features (and beyond). In: 14th International Conference on Learning
Representations. Vol 2026. OpenReview; 2026:145275-145314.'
apa: 'Iurada, L., Bombari, S., Tommasi, T., & Mondelli, M. (2026). A law of
data reconstruction for random features (and beyond). In 14th International
Conference on Learning Representations (Vol. 2026, pp. 145275–145314). Rio
de Janeiro, Brazil: OpenReview.'
chicago: Iurada, Leonardo, Simone Bombari, Tatiana Tommasi, and Marco Mondelli.
“A Law of Data Reconstruction for Random Features (and Beyond).” In 14th International
Conference on Learning Representations, 2026:145275–314. OpenReview, 2026.
ieee: L. Iurada, S. Bombari, T. Tommasi, and M. Mondelli, “A law of data reconstruction
for random features (and beyond),” in 14th International Conference on Learning
Representations, Rio de Janeiro, Brazil, 2026, vol. 2026, pp. 145275–145314.
ista: 'Iurada L, Bombari S, Tommasi T, Mondelli M. 2026. A law of data reconstruction
for random features (and beyond). 14th International Conference on Learning Representations.
ICLR: International Conference on Learning Representations vol. 2026, 145275–145314.'
mla: Iurada, Leonardo, et al. “A Law of Data Reconstruction for Random Features
(and Beyond).” 14th International Conference on Learning Representations,
vol. 2026, OpenReview, 2026, pp. 145275–314.
short: L. Iurada, S. Bombari, T. Tommasi, M. Mondelli, in:, 14th International Conference
on Learning Representations, OpenReview, 2026, pp. 145275–145314.
conference:
end_date: 2026-04-27
location: Rio de Janeiro, Brazil
name: 'ICLR: International Conference on Learning Representations '
start_date: 2026-04-23
corr_author: '1'
das_tickbox: '0'
date_created: 2026-09-09T13:31:46Z
date_published: 2026-01-26T00:00:00Z
date_updated: 2026-09-21T13:07:01Z
day: '26'
ddc:
- '000'
department:
- _id: GradSch
- _id: MaMo
external_id:
arxiv:
- '2509.22214'
file:
- access_level: open_access
checksum: a3dcba6649b7496124fa0ae68bf2c4cb
content_type: application/pdf
creator: cchlebak
date_created: 2026-09-11T12:07:48Z
date_updated: 2026-09-11T12:07:48Z
file_id: '22907'
file_name: 2026_ICLR_Iurada.pdf
file_size: 7869510
relation: main_file
success: 1
file_date_updated: 2026-09-11T12:07:48Z
has_accepted_license: '1'
intvolume: ' 2026'
language:
- iso: eng
month: '01'
oa: 1
oa_version: Published Version
page: 145275-145314
project:
- _id: 911e6d1f-16d5-11f0-9cad-c5c68c6a1cdf
grant_number: '101161364'
name: 'Inference in High Dimensions: Light-speed Algorithms and Information Limits'
- _id: 92099302-16d5-11f0-9cad-f9a785f54fbd
name: 'Trustworthy Deep Learning Theory: Private Over-Parameterized Models and Robust
LLMs'
publication: 14th International Conference on Learning Representations
publication_identifier:
isbn:
- '9798331339678'
publication_status: published
publisher: OpenReview
quality_controlled: '1'
related_material:
record:
- id: '22857'
relation: dissertation_contains
status: public
status: public
title: A law of data reconstruction for random features (and beyond)
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: conference
user_id: 8b945eb4-e2f2-11eb-945a-df72226e66a9
volume: 2026
year: '2026'
...
---
OA_place: publisher
_id: '22857'
abstract:
- lang: eng
text: "Artificial intelligence and machine learning have undergone an unprecedented
evolution in the past decade, motivating a research effort toward a theory able
to capture the qualitative behavior of large-scale neural systems. A central puzzle
has been the clear benefit of scaling architecture size and overfitting the training
set in supervised learning tasks. This evidence, in apparent contradiction with
classical statistical learning theory, pushed researchers to develop a new theory
capturing the interplay between the algorithmic and architectural bias of training
and the specific target function, differently from previous methods rooted in
uniform stability.\r\nThis approach has enabled a grounded understanding of novel
learning regimes, typically through formal limits where the number of training
samples $n$, data dimensions $d$, and model parameters $p$ grow to infinity at
different rates. \\\\\r\nIn this thesis, we follow this approach, focusing on
the trustworthiness of high-dimensional models: properties that are difficult
to control during training or deployment and often emerge under unpredictable
or adversarial conditions. In such settings, it is crucial to formally ensure
a priori the reliability of machine learning systems.\r\nFirst, we study data
memorization, both as label fitting and as the storage of private information
about training samples in trained parameters. We prove that $p = \\Omega(n)$ parameters
are sufficient for a deep neural network to memorize a generic set of labels,
and for a model to memorize spurious features across training data. We then give
evidence that $p = \\Omega(dn)$ parameters are instead necessary for an adversary
to reconstruct the full training set from the trained parameters.\r\nSecond, we
study robustness, both to adversarial perturbations and to distribution shift.
We first prove that $p = \\Omega(dn)$ parameters can be sufficient for a class
of neural networks to overfit the training data while guaranteeing robustness
to adversarial perturbations. Then, we focus on spurious correlations learning
in high-dimensional regression, studying the effect of the ridge regularization
parameter in the proportional regime $n = \\Theta(d)$, and connecting it via an
equivalence argument to the role of over-parameterization $p = \\Omega(n)$ in
neural networks. We also investigate the architectural bias of attention-based
networks, showing that they are sensitive to the replacement of individual words
in an embedded sentence, allowing them to generalize on sentences where the contextual
meaning depends on one or few words.\r\nFinally, we study differentially private
optimization in high-dimensional regimes. We prove that standard private gradient
methods do not suffer in the over-parameterized regime $p = \\Omega(n)$, challenging
the current wisdom based on stability-derived generalization bounds. We then consider
linear regression in the proportional regime $n = \\Theta(d)$, showing that standard
private gradient descent can achieve optimal rates under appropriate hyper-parameter
scaling, such as sufficiently small gradient clipping constants, whose role is
still debated in practice."
acknowledged_ssus:
- _id: ScienComp
acknowledgement: "This project was partially supported by the 2019 Lopez-Loreta prize,\r\nthe
European Union (ERC, INF2\r\n, project number 101161364), the Austrian Science Fund\r\n(FWF)
10.55776/COE12, and a Google PhD fellowship in machine intelligence. Furthermore,\r\nthe
candidate acknowledges the support from the Scientific Service Units of the Institute
of\r\nScience and Technology Austria through resources provided by Scientific Computing."
alternative_title:
- ISTA Thesis
article_processing_charge: No
author:
- first_name: Simone
full_name: Bombari, Simone
id: ca726dda-de17-11ea-bc14-f9da834f63aa
last_name: Bombari
citation:
ama: Bombari S. Trustworthy machine learning in high dimensions. 2026. doi:10.15479/AT-ISTA-22857
apa: Bombari, S. (2026). Trustworthy machine learning in high dimensions.
Institute of Science and Technology Austria. https://doi.org/10.15479/AT-ISTA-22857
chicago: Bombari, Simone. “Trustworthy Machine Learning in High Dimensions.” Institute
of Science and Technology Austria, 2026. https://doi.org/10.15479/AT-ISTA-22857.
ieee: S. Bombari, “Trustworthy machine learning in high dimensions,” Institute of
Science and Technology Austria, 2026.
ista: Bombari S. 2026. Trustworthy machine learning in high dimensions. Institute
of Science and Technology Austria.
mla: Bombari, Simone. Trustworthy Machine Learning in High Dimensions. Institute
of Science and Technology Austria, 2026, doi:10.15479/AT-ISTA-22857.
short: S. Bombari, Trustworthy Machine Learning in High Dimensions, Institute of
Science and Technology Austria, 2026.
corr_author: '1'
das_tickbox: '0'
date_created: 2026-09-08T13:40:08Z
date_published: 2026-09-08T00:00:00Z
date_updated: 2026-10-16T10:42:36Z
day: '08'
ddc:
- '519'
degree_awarded: PhD
department:
- _id: GradSch
- _id: MaMo
doi: 10.15479/AT-ISTA-22857
doi_confirm: '1'
file:
- access_level: closed
checksum: 8eab99d6dc6e4476826bdfc7c00fdebb
content_type: application/zip
creator: sbombari
date_created: 2026-09-08T13:28:38Z
date_updated: 2026-09-08T13:28:38Z
file_id: '22859'
file_name: Thesis copy.zip
file_size: 99154325
relation: source_file
- access_level: open_access
checksum: 007033bafe4622ff4c2e758301354df1
content_type: application/pdf
creator: sbombari
date_created: 2026-09-10T10:03:48Z
date_updated: 2026-09-10T10:03:48Z
file_id: '22898'
file_name: 2026_Bombari_Simone_Thesis.pdf
file_size: 12856211
relation: main_file
success: 1
file_date_updated: 2026-09-10T10:03:48Z
fulldoi: https://doi.org/10.15479/AT-ISTA-22857
has_accepted_license: '1'
keyword:
- machine learning
- high-dimensional statistics
- deep learning theory
- privacy
- memorization
- robustness
language:
- iso: eng
month: '09'
oa: 1
oa_version: Published Version
page: '446'
project:
- _id: 92099302-16d5-11f0-9cad-f9a785f54fbd
name: 'Trustworthy Deep Learning Theory: Private Over-Parameterized Models and Robust
LLMs'
- _id: 911e6d1f-16d5-11f0-9cad-c5c68c6a1cdf
grant_number: '101161364'
name: 'Inference in High Dimensions: Light-speed Algorithms and Information Limits'
- _id: 74caaef7-b034-11f1-8f2d-e0e993bb422e
grant_number: COE12
name: Bilateral Artificial Intelligence (Mondelli)
publication_identifier:
isbn:
- 978-3-99078-091-6
issn:
- 2663-337X
publication_status: published
publisher: Institute of Science and Technology Austria
related_material:
record:
- id: '12537'
relation: part_of_dissertation
status: public
- id: '21324'
relation: part_of_dissertation
status: public
- id: '22894'
relation: part_of_dissertation
status: public
- id: '19627'
relation: part_of_dissertation
status: public
- id: '12859'
relation: part_of_dissertation
status: public
- id: '18973'
relation: part_of_dissertation
status: public
- id: '18972'
relation: part_of_dissertation
status: public
researchdata_availability: no
status: public
supervisor:
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
supplementarymaterial: no
title: Trustworthy machine learning in high dimensions
type: dissertation
user_id: 8b945eb4-e2f2-11eb-945a-df72226e66a9
year: '2026'
...
---
OA_place: publisher
_id: '22258'
abstract:
- lang: eng
text: "Uncovering the genetic architecture of complex traits and pinpointing causal
molecular drivers require the ability to distinguish true signals from noise within
massive, high-dimensional omics datasets. To extract meaningful biological insights
from these datasets, such as identifying causal genetic variants and proteins,
scalable and accurate inference methods are essential. To this end, this thesis
develops novel Bayesian inference frameworks based on Vector Approximate Message
Passing and demonstrates their effectiveness in the modeling of disease onset
times and quantitative physical and clinical measures.\r\n\r\nFirst, we introduce
gVAMP, a Bayesian framework tailored for Genome-Wide Association Studies that
enables the joint modeling of quantitative complex traits across millions of genetic
variants. gVAMP demonstrates superior accuracy in variable selection and out-of-sample
polygenic risk prediction compared to state-of-the-art approaches. We model human
height using 17 million whole-genome sequence variants from the UK Biobank, incorporating
a vast number of rare variants and revealing novel associations. gVAMP achieves
a prediction accuracy of approximately 46% for human height, representing the
highest reported performance for this trait to date. \r\n\r\nSecond, we present
vampW, a Bayesian framework for survival analysis applied to proteomic data. By
effectively handling right-censoring and complex protein dependencies within the
UK Biobank Pharma Proteomics Project dataset, vampW identifies 219 protein associations
across 24 disease outcomes, the majority of which are not among the top marginal
discoveries. We further adjust protein levels for exponential age effects, yielding
1,308 associations and highlighting the sensitivity of the analysis to the chosen
age-correction methodology. Finally, vampW improves upon the variable selection
capabilities of the commonly used (penalized) variants of the Cox proportional
hazards model and delivers state-of-the-art out-of-sample prediction of disease
onset times.\r\n\r\nCollectively, these methods provide powerful tools for dissecting
the genetic architecture of complex traits and the proteomic drivers of disease
onset. Furthermore, by delivering accurate polygenic risk scores and precise predictions
of onset times, this work advances the capabilities of personalized medicine and
clinical risk stratification."
acknowledged_ssus:
- _id: ScienComp
acknowledgement: "This work was supported in part by the Swiss National Science Foundation
through the\r\nEccellenza Grant \"Improving estimation and prediction of common
complex disease risk\"\r\n(grant number PCEGP3_181181); the European Research Council
through the grant\r\n\"Inference in High Dimensions: Light-speed Algorithms and
Information Limits\" (grant\r\nnumber 101161364); and the Fondation Jean-Jacques
et Felicia Lopez-Loreta through the\r\nPrix Lopez-Loretta 2019.\r\n"
alternative_title:
- ISTA Thesis
article_processing_charge: No
author:
- first_name: Al
full_name: Depope, Al
id: 0b77531d-dbcd-11ea-9d1d-a8eee0bf3830
last_name: Depope
citation:
ama: 'Depope A. From sparse selection to risk prediction: Approximate message passing
for proteomic survival models and large-scale genomics. 2026. doi:10.15479/AT-ISTA-22258'
apa: 'Depope, A. (2026). From sparse selection to risk prediction: Approximate
message passing for proteomic survival models and large-scale genomics. Institute
of Science and Technology Austria. https://doi.org/10.15479/AT-ISTA-22258'
chicago: 'Depope, Al. “From Sparse Selection to Risk Prediction: Approximate Message
Passing for Proteomic Survival Models and Large-Scale Genomics.” Institute of
Science and Technology Austria, 2026. https://doi.org/10.15479/AT-ISTA-22258.'
ieee: 'A. Depope, “From sparse selection to risk prediction: Approximate message
passing for proteomic survival models and large-scale genomics,” Institute of
Science and Technology Austria, 2026.'
ista: 'Depope A. 2026. From sparse selection to risk prediction: Approximate message
passing for proteomic survival models and large-scale genomics. Institute of Science
and Technology Austria.'
mla: 'Depope, Al. From Sparse Selection to Risk Prediction: Approximate Message
Passing for Proteomic Survival Models and Large-Scale Genomics. Institute
of Science and Technology Austria, 2026, doi:10.15479/AT-ISTA-22258.'
short: 'A. Depope, From Sparse Selection to Risk Prediction: Approximate Message
Passing for Proteomic Survival Models and Large-Scale Genomics, Institute of Science
and Technology Austria, 2026.'
corr_author: '1'
das_tickbox: '1'
date_created: 2026-07-10T13:27:20Z
date_published: 2026-07-11T00:00:00Z
date_updated: 2026-10-20T09:03:21Z
day: '11'
ddc:
- '576'
- '610'
- '006'
degree_awarded: PhD
department:
- _id: GradSch
- _id: MaRo
- _id: MaMo
doi: 10.15479/AT-ISTA-22258
doi_confirm: '1'
file:
- access_level: open_access
checksum: 9ab386790515628d957a194f30a7ccb4
content_type: application/pdf
creator: adepope
date_created: 2026-07-13T14:52:19Z
date_updated: 2026-07-13T14:52:19Z
file_id: '22316'
file_name: 2026_Depope_Al_Thesis.pdf
file_size: 25109878
relation: main_file
- access_level: closed
checksum: 8ed8fb63f76a695d5b6fec35343f4b90
content_type: application/zip
creator: adepope
date_created: 2026-07-13T14:56:41Z
date_updated: 2026-07-13T14:56:41Z
file_id: '22317'
file_name: 2026_Depope_Al_Thesis.zip
file_size: 1203199939
relation: source_file
file_date_updated: 2026-07-13T14:56:41Z
fulldoi: https://doi.org/10.15479/AT-ISTA-22258
has_accepted_license: '1'
keyword:
- Approximate Message Passing
- GWAS
- Genomics
- Proteomics
- Survival modeling
language:
- iso: eng
month: '07'
oa: 1
oa_version: Published Version
page: '169'
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
- _id: 911e6d1f-16d5-11f0-9cad-c5c68c6a1cdf
grant_number: '101161364'
name: 'Inference in High Dimensions: Light-speed Algorithms and Information Limits'
- _id: 9B8D11D6-BA93-11EA-9121-9846C619BF3A
grant_number: PCEGP3_181181
name: Improving estimation and prediction of common complex disease risk
publication_identifier:
issn:
- 2663-337X
publication_status: published
publisher: Institute of Science and Technology Austria
related_material:
record:
- id: '21488'
relation: part_of_dissertation
status: public
status: public
supervisor:
- first_name: Matthew Richard
full_name: Robinson, Matthew Richard
id: E5D42276-F5DA-11E9-8E24-6303E6697425
last_name: Robinson
orcid: 0000-0001-8982-8813
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
title: 'From sparse selection to risk prediction: Approximate message passing for
proteomic survival models and large-scale genomics'
type: dissertation
user_id: 8b945eb4-e2f2-11eb-945a-df72226e66a9
year: '2026'
...
---
DOAJ_listed: '1'
OA_place: publisher
OA_type: gold
_id: '21488'
abstract:
- lang: eng
text: Human height is a model for the genetic analysis of complex traits, and recent
studies suggest the presence of thousands of common genetic variant associations
and hundreds of low-frequency/rare variants. Here, we develop a new algorithmic
paradigm based on approximate message passing (genomic vector approximate message
passing [gVAMP]) for identifying DNA sequence variants associated with complex
traits and common diseases in large-scale whole-genome sequencing (WGS) data.
We show that gVAMP accurately localizes associations to variants with the correct
frequency and position in the DNA, outperforming existing fine-mapping methods
in selecting the appropriate genetic variants within WGS data. We then apply gVAMP
to jointly model the relationship of tens of millions of WGS variants with human
height in hundreds of thousands of UK Biobank individuals. We identify 59 rare
variants and gene burden scores alongside many hundreds of DNA regions containing
common variant associations and show that understanding the genetic basis of complex
traits will require the joint analysis of hundreds of millions of variables measured
on millions of people. The polygenic risk scores obtained from gVAMP have high
accuracy (including a prediction accuracy of ∼46% for human height) and outperform
current methods for downstream tasks such as mixed linear model association testing
across 13 UK Biobank traits. In conclusion, gVAMP offers a scalable foundation
for a wider range of analyses in WGS data.
acknowledgement: We thank Malgorzata Borczyk for creating the gene burden scores.
We thank Robin Beaumont, Amedeo Roberto Esposito, Gareth Hawkes, Philip Schniter,
Matthew Stephens, Pragya Sur, Peter Visscher, Michael Weedon, and Harry Wright for
providing valuable suggestions and comments on earlier versions of the work. This
project was funded by a Lopez-Loreta Prize to M.M., an SNSF Eccellenza Grant to
M.R.R. (PCEGP3-181181), an ERC Starting Grant to M.M. (INF2, project number 101161364),
and core funding from ISTA. High-performance computing was supported by the Scientific
Service Units (SSU) of ISTA through resources provided by Scientific Computing (SciComp).
We would like to acknowledge the participants and investigators of the UK Biobank
study. We gratefully acknowledge the All of Us participants for their contributions,
without whom this research would not have been possible. We also thank the National
Institutes of Health All of Us Research Program for making available the participant
data (and/or samples and/or cohort) examined in this study.
article_number: '101162'
article_processing_charge: Yes
article_type: original
author:
- first_name: Al
full_name: Depope, Al
id: 0b77531d-dbcd-11ea-9d1d-a8eee0bf3830
last_name: Depope
- first_name: Jakub
full_name: Bajzik, Jakub
id: b995e25b-8c4b-11ed-a6d8-f71b7bcd6122
last_name: Bajzik
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Matthew Richard
full_name: Robinson, Matthew Richard
id: E5D42276-F5DA-11E9-8E24-6303E6697425
last_name: Robinson
orcid: 0000-0001-8982-8813
citation:
ama: Depope A, Bajzik J, Mondelli M, Robinson MR. Joint modeling of whole-genome
sequencing data for human height via approximate message passing. Cell Genomics.
2026;6(5). doi:10.1016/j.xgen.2026.101162
apa: Depope, A., Bajzik, J., Mondelli, M., & Robinson, M. R. (2026). Joint modeling
of whole-genome sequencing data for human height via approximate message passing.
Cell Genomics. Cell Press. https://doi.org/10.1016/j.xgen.2026.101162
chicago: Depope, Al, Jakub Bajzik, Marco Mondelli, and Matthew Richard Robinson.
“Joint Modeling of Whole-Genome Sequencing Data for Human Height via Approximate
Message Passing.” Cell Genomics. Cell Press, 2026. https://doi.org/10.1016/j.xgen.2026.101162.
ieee: A. Depope, J. Bajzik, M. Mondelli, and M. R. Robinson, “Joint modeling of
whole-genome sequencing data for human height via approximate message passing,”
Cell Genomics, vol. 6, no. 5. Cell Press, 2026.
ista: Depope A, Bajzik J, Mondelli M, Robinson MR. 2026. Joint modeling of whole-genome
sequencing data for human height via approximate message passing. Cell Genomics.
6(5), 101162.
mla: Depope, Al, et al. “Joint Modeling of Whole-Genome Sequencing Data for Human
Height via Approximate Message Passing.” Cell Genomics, vol. 6, no. 5,
101162, Cell Press, 2026, doi:10.1016/j.xgen.2026.101162.
short: A. Depope, J. Bajzik, M. Mondelli, M.R. Robinson, Cell Genomics 6 (2026).
corr_author: '1'
das_tickbox: '1'
dataavailabilitystatement: "This project uses the UK Biobank data under project number
35520. UK Biobank genotypic and phenotypic data are available through a formal request
at http://www.ukbiobank.ac.uk. It also uses genotypic and phenotypic data from the
All of Us study, which are also available through a formal request at https://www.researchallofus.org/data-tools/data-access/.
All summary statistic estimates are released publicly on Dryad: https://doi.org/10.5061/dryad.cz8w9gjjc.\r\n•\r\nThe
gVAMP code developed in this work is open source and has been deposited on GitHub,
where it is publicly available at https://github.com/medical-genomics-group/gVAMP,
and the code used to generate the data in the manuscript are available from Zenodo
https://doi.org/10.5281/zenodo.17935521. The URLs of other software used are listed
in the key resources table."
date_created: 2026-03-23T15:10:03Z
date_published: 2026-05-13T00:00:00Z
date_updated: 2026-10-20T09:03:22Z
day: '13'
ddc:
- '000'
- '570'
department:
- _id: MaMo
- _id: MaRo
doi: 10.1016/j.xgen.2026.101162
external_id:
pmid:
- '41713425'
file:
- access_level: open_access
checksum: 6b59686f8d9733add4f23d23f3dd9df0
content_type: application/pdf
creator: dernst
date_created: 2026-07-28T07:06:26Z
date_updated: 2026-07-28T07:06:26Z
file_id: '22596'
file_name: 2026_CellGenomics_Depope.pdf
file_size: 3736705
relation: main_file
success: 1
file_date_updated: 2026-07-28T07:06:26Z
fulldoi: https://doi.org/10.1016/j.xgen.2026.101162
has_accepted_license: '1'
intvolume: ' 6'
issue: '5'
language:
- iso: eng
license: https://creativecommons.org/licenses/by-nc-nd/4.0/
month: '05'
oa: 1
oa_version: Published Version
pmid: 1
project:
- _id: 059876FA-7A3F-11EA-A408-12923DDC885E
name: Prix Lopez-Loretta 2019 - Marco Mondelli
- _id: 911e6d1f-16d5-11f0-9cad-c5c68c6a1cdf
grant_number: '101161364'
name: 'Inference in High Dimensions: Light-speed Algorithms and Information Limits'
- _id: 9B8D11D6-BA93-11EA-9121-9846C619BF3A
grant_number: PCEGP3_181181
name: Improving estimation and prediction of common complex disease risk
publication: Cell Genomics
publication_identifier:
eissn:
- 2666-979X
publication_status: published
publisher: Cell Press
quality_controlled: '1'
related_material:
link:
- description: News on ISTA website
relation: press_release
url: https://ista.ac.at/en/news/big-data-and-human-height/
record:
- id: '22258'
relation: dissertation_contains
status: public
researchdata_availability: yes
scopus_import: '1'
status: public
supplementarymaterial: yes
title: Joint modeling of whole-genome sequencing data for human height via approximate
message passing
tmp:
image: /images/cc_by_nc_nd.png
legal_code_url: https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode
name: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International
(CC BY-NC-ND 4.0)
short: CC BY-NC-ND (4.0)
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 6
year: '2026'
...
---
OA_place: publisher
OA_type: gold
_id: '21325'
abstract:
- lang: eng
text: Test-time training (TTT) methods explicitly update the weights of a model
to adapt to the specific test instance, and they have found success in a variety
of settings, including most recently language modeling and reasoning. To demystify
this success, we investigate a gradient-based TTT algorithm for in-context learning,
where we train a transformer model on the in-context demonstrations provided in
the test prompt. Specifically, we provide a comprehensive theoretical characterization
of linear transformers when the update rule is a single gradient step. Our theory
(i) delineates the role of alignment between pretraining distribution and target
task, (ii) demystifies how TTT can alleviate distribution shift, and (iii) quantifies
the sample complexity of TTT including how it can significantly reduce the eventual
sample size required for in-context learning. As our empirical contribution, we
study the benefits of TTT for TabPFN, a tabular foundation model. In line with
our theory, we demonstrate that TTT significantly reduces the required sample
size for tabular classification (3 to 5 times fewer) unlocking substantial inference
efficiency with a negligible training cost.
acknowledgement: "H.A.G., M.E.I., X.Z., and S.O. were supported in part by the NSF
grants CCF2046816, CCF-2403075, CCF-2008020, and the Office of Naval Research grant
N000142412289.\r\nM. M. is funded by the European Union (ERC, INF2 , project number
101161364). Views and opinions expressed are, however, those of the author(s) only
and do not necessarily\r\nreflect those of the European Union or the European Research
Council Executive Agency. Neither the European Union nor the granting authority
can be held responsible for them. M.S. is supported by the Packard Fellowship in
Science and Engineering, a Sloan Research Fellowship in Mathematics, an NSF-CAREER
under award #1846369, DARPA FastNICS program, and NSF-CIF awards #1813877 and #2008443,
and NIH DP2LM014564-01. The authors also\r\nacknowledge further support from Open
Philanthropy, OpenAI, Amazon Research, Google Research, and Microsoft Research."
alternative_title:
- PMLR
article_processing_charge: No
author:
- first_name: Halil Alperen
full_name: Gozeten, Halil Alperen
last_name: Gozeten
- first_name: Muhammed Emrullah
full_name: Ildiz, Muhammed Emrullah
last_name: Ildiz
- first_name: Xuechen
full_name: Zhang, Xuechen
last_name: Zhang
- first_name: Mahdi
full_name: Soltanolkotabi, Mahdi
last_name: Soltanolkotabi
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Samet
full_name: Oymak, Samet
last_name: Oymak
citation:
ama: 'Gozeten HA, Ildiz ME, Zhang X, Soltanolkotabi M, Mondelli M, Oymak S. Test-time
training provably improves transformers as in-context learners. In: Proceedings
of the 42nd International Conference on Machine Learning. Vol 267. ML Research
Press; 2025:20266-20295.'
apa: 'Gozeten, H. A., Ildiz, M. E., Zhang, X., Soltanolkotabi, M., Mondelli, M.,
& Oymak, S. (2025). Test-time training provably improves transformers as in-context
learners. In Proceedings of the 42nd International Conference on Machine Learning
(Vol. 267, pp. 20266–20295). Vancouver, Canada: ML Research Press.'
chicago: Gozeten, Halil Alperen, Muhammed Emrullah Ildiz, Xuechen Zhang, Mahdi Soltanolkotabi,
Marco Mondelli, and Samet Oymak. “Test-Time Training Provably Improves Transformers
as in-Context Learners.” In Proceedings of the 42nd International Conference
on Machine Learning, 267:20266–95. ML Research Press, 2025.
ieee: H. A. Gozeten, M. E. Ildiz, X. Zhang, M. Soltanolkotabi, M. Mondelli, and
S. Oymak, “Test-time training provably improves transformers as in-context learners,”
in Proceedings of the 42nd International Conference on Machine Learning,
Vancouver, Canada, 2025, vol. 267, pp. 20266–20295.
ista: 'Gozeten HA, Ildiz ME, Zhang X, Soltanolkotabi M, Mondelli M, Oymak S. 2025.
Test-time training provably improves transformers as in-context learners. Proceedings
of the 42nd International Conference on Machine Learning. ICML: International
Conference on Machine Learning, PMLR, vol. 267, 20266–20295.'
mla: Gozeten, Halil Alperen, et al. “Test-Time Training Provably Improves Transformers
as in-Context Learners.” Proceedings of the 42nd International Conference on
Machine Learning, vol. 267, ML Research Press, 2025, pp. 20266–95.
short: H.A. Gozeten, M.E. Ildiz, X. Zhang, M. Soltanolkotabi, M. Mondelli, S. Oymak,
in:, Proceedings of the 42nd International Conference on Machine Learning, ML
Research Press, 2025, pp. 20266–20295.
conference:
end_date: 2025-07-19
location: Vancouver, Canada
name: 'ICML: International Conference on Machine Learning'
start_date: 2025-07-13
date_created: 2026-02-18T12:00:44Z
date_published: 2025-11-30T00:00:00Z
date_updated: 2026-02-19T08:18:24Z
day: '30'
ddc:
- '000'
department:
- _id: MaMo
external_id:
pmid:
- '41321376'
file:
- access_level: open_access
checksum: f774f8619a0d72f3975d9cb23942a1e9
content_type: application/pdf
creator: dernst
date_created: 2026-02-19T08:15:48Z
date_updated: 2026-02-19T08:15:48Z
file_id: '21336'
file_name: 2025_ICML_Gozeten.pdf
file_size: 471176
relation: main_file
success: 1
file_date_updated: 2026-02-19T08:15:48Z
has_accepted_license: '1'
intvolume: ' 267'
language:
- iso: eng
month: '11'
oa: 1
oa_version: Published Version
page: 20266-20295
pmid: 1
project:
- _id: 911e6d1f-16d5-11f0-9cad-c5c68c6a1cdf
grant_number: '101161364'
name: 'Inference in High Dimensions: Light-speed Algorithms and Information Limits'
publication: Proceedings of the 42nd International Conference on Machine Learning
publication_identifier:
eissn:
- 2640-3498
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
status: public
title: Test-time training provably improves transformers as in-context learners
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 267
year: '2025'
...
---
OA_place: publisher
OA_type: gold
_id: '21328'
abstract:
- lang: eng
text: Multi-index models provide a popular framework to investigate the learnability
of functions with low-dimensional structure and, also due to their connections
with neural networks, they have been object of recent intensive study. In this
paper, we focus on recovering the subspace spanned by the signals via spectral
estimators – a family of methods routinely used in practice, often as a warm-start
for iterative algorithms. Our main technical contribution is a precise asymptotic
characterization of the performance of spectral methods, when sample size and
input dimension grow proportionally and the dimension p of the space to recover
is fixed. Specifically, we locate the top-p eigenvalues of the spectral matrix
and establish the overlaps between the corresponding eigenvectors (which give
the spectral estimators) and a basis of the signal subspace. Our analysis unveils
a phase transition phenomenon in which, as the sample complexity grows, eigenvalues
escape from the bulk of the spectrum and, when that happens, eigenvectors recover
directions of the desired subspace. The precise characterization we put forward
enables the optimization of the data preprocessing, thus allowing to identify
the spectral estimator that requires the minimal sample size for weak recovery.
acknowledgement: "This work was done when Y. Z. was at the Institute of Science and
Technology Austria. Y. Z. and\r\nM. M. are funded by the European Union (ERC, INF2,
project number 101161364). Views and\r\nopinions expressed are however those of
the author(s) only and do not necessarily reflect those of the European Union or
the European Research Council Executive Agency. Neither the European Union nor the
granting authority can be held responsible for them. The authors would like to acknowledge
(in alphabetical order) discussions with Yatin Dandi, Leonardo Defilippis and Bruno
Loureiro concerning their parallel work (Defilippis et al., 2025)."
alternative_title:
- PMLR
article_processing_charge: No
arxiv: 1
author:
- first_name: Filip
full_name: Kovačević, Filip
id: d0258e7b-50b8-11ef-ad56-8b9f537b6b1b
last_name: Kovačević
- first_name: Zhang
full_name: Yihan, Zhang
last_name: Yihan
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: 'Kovačević F, Yihan Z, Mondelli M. Spectral estimators for multi-index models:
Precise asymptotics and optimal weak recovery. In: Proceedings of 38th Conference
on Learning Theory. Vol 291. ML Research Press; 2025:3354-3404.'
apa: 'Kovačević, F., Yihan, Z., & Mondelli, M. (2025). Spectral estimators for
multi-index models: Precise asymptotics and optimal weak recovery. In Proceedings
of 38th Conference on Learning Theory (Vol. 291, pp. 3354–3404). Lyon, France:
ML Research Press.'
chicago: 'Kovačević, Filip, Zhang Yihan, and Marco Mondelli. “Spectral Estimators
for Multi-Index Models: Precise Asymptotics and Optimal Weak Recovery.” In Proceedings
of 38th Conference on Learning Theory, 291:3354–3404. ML Research Press, 2025.'
ieee: 'F. Kovačević, Z. Yihan, and M. Mondelli, “Spectral estimators for multi-index
models: Precise asymptotics and optimal weak recovery,” in Proceedings of 38th
Conference on Learning Theory, Lyon, France, 2025, vol. 291, pp. 3354–3404.'
ista: 'Kovačević F, Yihan Z, Mondelli M. 2025. Spectral estimators for multi-index
models: Precise asymptotics and optimal weak recovery. Proceedings of 38th Conference
on Learning Theory. COLT: Conference on Learning Theory, PMLR, vol. 291, 3354–3404.'
mla: 'Kovačević, Filip, et al. “Spectral Estimators for Multi-Index Models: Precise
Asymptotics and Optimal Weak Recovery.” Proceedings of 38th Conference on Learning
Theory, vol. 291, ML Research Press, 2025, pp. 3354–404.'
short: F. Kovačević, Z. Yihan, M. Mondelli, in:, Proceedings of 38th Conference
on Learning Theory, ML Research Press, 2025, pp. 3354–3404.
conference:
end_date: 2025-07-04
location: Lyon, France
name: 'COLT: Conference on Learning Theory'
start_date: 2025-06-30
corr_author: '1'
date_created: 2026-02-18T12:12:47Z
date_published: 2025-07-01T00:00:00Z
date_updated: 2026-02-19T09:03:53Z
day: '01'
ddc:
- '000'
department:
- _id: MaMo
external_id:
arxiv:
- '2502.01583'
file:
- access_level: open_access
checksum: 19aa70ab4f57fb9067b6ebb99a5fd6f0
content_type: application/pdf
creator: dernst
date_created: 2026-02-19T09:03:43Z
date_updated: 2026-02-19T09:03:43Z
file_id: '21339'
file_name: 2025_LearningTheory_Kovacevic.pdf
file_size: 844611
relation: main_file
success: 1
file_date_updated: 2026-02-19T09:03:43Z
has_accepted_license: '1'
intvolume: ' 291'
language:
- iso: eng
month: '07'
oa: 1
oa_version: Published Version
page: 3354-3404
project:
- _id: 911e6d1f-16d5-11f0-9cad-c5c68c6a1cdf
grant_number: '101161364'
name: 'Inference in High Dimensions: Light-speed Algorithms and Information Limits'
publication: Proceedings of 38th Conference on Learning Theory
publication_identifier:
eissn:
- 2640-3498
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'Spectral estimators for multi-index models: Precise asymptotics and optimal
weak recovery'
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 291
year: '2025'
...
---
OA_place: publisher
OA_type: diamond
_id: '20033'
abstract:
- lang: eng
text: 'A growing number of machine learning scenarios rely on knowledge distillation
where one uses the output of a surrogate model as labels to supervise the training
of a target model. In this work, we provide a sharp characterization of this process
for ridgeless, high-dimensional regression, under two settings: (i) model shift,
where the surrogate model is arbitrary, and (ii) distribution shift, where the
surrogate model is the solution of empirical risk minimization with out-of-distribution
data. In both cases, we characterize the precise risk of the target model through
non-asymptotic bounds in terms of sample size and data distribution under mild
conditions. As a consequence, we identify the form of the optimal surrogate model,
which reveals the benefits and limitations of discarding weak features in a data-dependent
fashion. In the context of weak-to-strong (W2S) generalization, this has the interpretation
that (i) W2S training, with the surrogate as the weak model, can provably outperform
training with strong labels under the same data budget, but (ii) it is unable
to improve the data scaling law. We validate our results on numerical experiments
both on ridgeless regression and on neural network architectures.'
acknowledgement: M.E.I., H.A.G., E.O.T., S.O. are supported by the NSF grants CCF-2046816,
CCF-2403075, the Office of Naval Research grant N000142412289, an OpenAI Agentic
AI Systems grant, and gifts by Open Philanthropy and Google Research. M. M. is funded
by the European Union (ERC, INF2, project number 101161364). Views and opinions
expressed are however those of the author(s) only and do not necessarily reflect
those of the European Union or the European Research Council Executive Agency. Neither
the European Union nor the granting authority can be held responsible for them.
article_processing_charge: No
arxiv: 1
author:
- first_name: M.
full_name: Emrullah Ildiz, M.
last_name: Emrullah Ildiz
- first_name: Halil Alperen
full_name: Gozeten, Halil Alperen
last_name: Gozeten
- first_name: Ege Onur
full_name: Taga, Ege Onur
last_name: Taga
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
- first_name: Samet
full_name: Oymak, Samet
last_name: Oymak
citation:
ama: 'Emrullah Ildiz M, Gozeten HA, Taga EO, Mondelli M, Oymak S. High-dimensional
analysis of knowledge distillation: Weak-to-Strong generalization and scaling
laws. In: 13th International Conference on Learning Representations. ICLR;
2025:2967-3006.'
apa: 'Emrullah Ildiz, M., Gozeten, H. A., Taga, E. O., Mondelli, M., & Oymak,
S. (2025). High-dimensional analysis of knowledge distillation: Weak-to-Strong
generalization and scaling laws. In 13th International Conference on Learning
Representations (pp. 2967–3006). Singapore, Singapore: ICLR.'
chicago: 'Emrullah Ildiz, M., Halil Alperen Gozeten, Ege Onur Taga, Marco Mondelli,
and Samet Oymak. “High-Dimensional Analysis of Knowledge Distillation: Weak-to-Strong
Generalization and Scaling Laws.” In 13th International Conference on Learning
Representations, 2967–3006. ICLR, 2025.'
ieee: 'M. Emrullah Ildiz, H. A. Gozeten, E. O. Taga, M. Mondelli, and S. Oymak,
“High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization
and scaling laws,” in 13th International Conference on Learning Representations,
Singapore, Singapore, 2025, pp. 2967–3006.'
ista: 'Emrullah Ildiz M, Gozeten HA, Taga EO, Mondelli M, Oymak S. 2025. High-dimensional
analysis of knowledge distillation: Weak-to-Strong generalization and scaling
laws. 13th International Conference on Learning Representations. ICLR: International
Conference on Learning Representations, 2967–3006.'
mla: 'Emrullah Ildiz, M., et al. “High-Dimensional Analysis of Knowledge Distillation:
Weak-to-Strong Generalization and Scaling Laws.” 13th International Conference
on Learning Representations, ICLR, 2025, pp. 2967–3006.'
short: M. Emrullah Ildiz, H.A. Gozeten, E.O. Taga, M. Mondelli, S. Oymak, in:, 13th
International Conference on Learning Representations, ICLR, 2025, pp. 2967–3006.
conference:
end_date: 2025-04-28
location: Singapore, Singapore
name: 'ICLR: International Conference on Learning Representations'
start_date: 2025-04-24
date_created: 2025-07-20T22:02:02Z
date_published: 2025-04-01T00:00:00Z
date_updated: 2025-08-04T08:33:58Z
day: '01'
ddc:
- '000'
department:
- _id: MaMo
external_id:
arxiv:
- '2410.18837'
file:
- access_level: open_access
checksum: 5a38b093ebb4ee4eb662ea142621a5ca
content_type: application/pdf
creator: dernst
date_created: 2025-08-04T08:32:38Z
date_updated: 2025-08-04T08:32:38Z
file_id: '20112'
file_name: 2025_ICLR_Ildiz.pdf
file_size: 528171
relation: main_file
success: 1
file_date_updated: 2025-08-04T08:32:38Z
has_accepted_license: '1'
language:
- iso: eng
month: '04'
oa: 1
oa_version: Published Version
page: 2967-3006
project:
- _id: 911e6d1f-16d5-11f0-9cad-c5c68c6a1cdf
grant_number: '101161364'
name: 'Inference in High Dimensions: Light-speed Algorithms and Information Limits'
publication: 13th International Conference on Learning Representations
publication_identifier:
isbn:
- '9798331320850'
publication_status: published
publisher: ICLR
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'High-dimensional analysis of knowledge distillation: Weak-to-Strong generalization
and scaling laws'
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2025'
...
---
OA_place: publisher
OA_type: diamond
_id: '20035'
abstract:
- lang: eng
text: "Deep neural networks (DNNs) at convergence consistently represent the training
data in the last layer via a geometric structure referred to as neural collapse.
This empirical evidence has spurred a line of theoretical research aimed at proving
the emergence of neural collapse, mostly focusing on the unconstrained features
model. Here, the features of the penultimate layer are free variables, which makes
the model data-agnostic and puts into question its ability to capture DNN training.
Our work addresses the issue, moving away from unconstrained features and\r\nstudying
DNNs that end with at least two linear layers. We first prove generic guarantees
on neural collapse that assume (i) low training error and balancedness of linear
layers (for within-class variability collapse), and (ii) bounded conditioning
of the features before the linear part (for orthogonality of class-means, and
their alignment with weight matrices). The balancedness refers to the fact that
W⊤ℓ+1Wℓ+1 ≈ WℓW⊤ℓfor any pair of consecutive weight matrices of the linear part,
and the bounded conditioning requires a well-behaved ratio between largest and
smallest non-zero singular values of the features. We then show that such assumptions
hold for gradient descent training with weight decay: (i) for networks with a
wide first layer, we prove low training error and balancedness, and (ii) for solutions
that are either nearly optimal or stable under large learning rates, we additionally
prove the bounded conditioning. Taken together, our results are the first to show
neural collapse in the end-to-end training of DNNs."
acknowledgement: M. M. and P. S. are funded by the European Union (ERC, INF2, project
number 101161364). Views and opinions expressed are however those of the author(s)
only and do not necessarily reflect those of the European Union or the European
Research Council Executive Agency. Neither the European Union nor the granting authority
can be held responsible for them.
article_processing_charge: No
arxiv: 1
author:
- first_name: Arthur
full_name: Jacot, Arthur
last_name: Jacot
- first_name: Peter
full_name: Súkeník, Peter
id: d64d6a8d-eb8e-11eb-b029-96fd216dec3c
last_name: Súkeník
- first_name: Zihan
full_name: Wang, Zihan
last_name: Wang
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: 'Jacot A, Súkeník P, Wang Z, Mondelli M. Wide neural networks trained with
weight decay provably exhibit neural collapse. In: 13th International Conference
on Learning Representations. ICLR; 2025:1905-1931.'
apa: 'Jacot, A., Súkeník, P., Wang, Z., & Mondelli, M. (2025). Wide neural networks
trained with weight decay provably exhibit neural collapse. In 13th International
Conference on Learning Representations (pp. 1905–1931). Singapore, Singapore:
ICLR.'
chicago: Jacot, Arthur, Peter Súkeník, Zihan Wang, and Marco Mondelli. “Wide Neural
Networks Trained with Weight Decay Provably Exhibit Neural Collapse.” In 13th
International Conference on Learning Representations, 1905–31. ICLR, 2025.
ieee: A. Jacot, P. Súkeník, Z. Wang, and M. Mondelli, “Wide neural networks trained
with weight decay provably exhibit neural collapse,” in 13th International
Conference on Learning Representations, Singapore, Singapore, 2025, pp. 1905–1931.
ista: 'Jacot A, Súkeník P, Wang Z, Mondelli M. 2025. Wide neural networks trained
with weight decay provably exhibit neural collapse. 13th International Conference
on Learning Representations. ICLR: International Conference on Learning Representations,
1905–1931.'
mla: Jacot, Arthur, et al. “Wide Neural Networks Trained with Weight Decay Provably
Exhibit Neural Collapse.” 13th International Conference on Learning Representations,
ICLR, 2025, pp. 1905–31.
short: A. Jacot, P. Súkeník, Z. Wang, M. Mondelli, in:, 13th International Conference
on Learning Representations, ICLR, 2025, pp. 1905–1931.
conference:
end_date: 2025-04-28
location: Singapore, Singapore
name: 'ICLR: International Conference on Learning Representations'
start_date: 2025-04-24
corr_author: '1'
date_created: 2025-07-20T22:02:02Z
date_published: 2025-04-01T00:00:00Z
date_updated: 2025-08-04T08:47:00Z
day: '01'
ddc:
- '000'
department:
- _id: MaMo
external_id:
arxiv:
- '2410.04887'
file:
- access_level: open_access
checksum: 59c48c173887139647cc9839c0801136
content_type: application/pdf
creator: dernst
date_created: 2025-08-04T08:45:43Z
date_updated: 2025-08-04T08:45:43Z
file_id: '20114'
file_name: 2025_ICLR_Jacot.pdf
file_size: 1337236
relation: main_file
success: 1
file_date_updated: 2025-08-04T08:45:43Z
has_accepted_license: '1'
language:
- iso: eng
month: '04'
oa: 1
oa_version: Published Version
page: 1905-1931
project:
- _id: 911e6d1f-16d5-11f0-9cad-c5c68c6a1cdf
grant_number: '101161364'
name: 'Inference in High Dimensions: Light-speed Algorithms and Information Limits'
publication: 13th International Conference on Learning Representations
publication_identifier:
isbn:
- '9798331320850'
publication_status: published
publisher: ICLR
quality_controlled: '1'
scopus_import: '1'
status: public
title: Wide neural networks trained with weight decay provably exhibit neural collapse
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2025'
...
---
OA_place: publisher
OA_type: free access
_id: '22825'
abstract:
- lang: eng
text: "The empirical emergence of neural collapse—a surprising symmetry in the feature
representations of the training data in the penultimate layer of deep neural\r\nnetworks—has
spurred a line of theoretical research aimed at its understanding.\r\nHowever,
existing work focuses on data-agnostic models or, when data structure is\r\ntaken
into account, it remains limited to multi-layer perceptrons. Our paper fills\r\nboth
these gaps by analyzing modern architectures in a data-aware regime: we\r\nprove
that global optima of deep regularized transformers and residual networks\r\n(ResNets)
with LayerNorm trained with cross entropy or mean squared error loss\r\nare approximately
collapsed, and the approximation gets tighter as the depth grows.\r\nMore generally,
we formally reduce any end-to-end large-depth ResNet or transformer training into
an equivalent unconstrained features model, thus justifying its\r\nwide use in
the literature even beyond data-agnostic settings. Our theoretical results\r\nare
supported by experiments on computer vision and language datasets showing\r\nthat,
as the depth grows, neural collapse indeed becomes more prominent."
acknowledged_ssus:
- _id: ScienComp
acknowledgement: "M. M. and P. S. are funded by the European Union (ERC, INF2\r\n,
project number 101161364). Views\r\nand opinions expressed are however those of
the author(s) only and do not necessarily reflect those\r\nof the European Union
or the European Research Council Executive Agency. Neither the European\r\nUnion
nor the granting authority can be held responsible for them. This research was supported\r\nby
the Scientific Service Units (SSU) of ISTA through resources provided by Scientific
Computing (SciComp)."
alternative_title:
- Advances in Neural Information Processing Systems
article_processing_charge: No
arxiv: 1
author:
- first_name: Peter
full_name: Súkeník, Peter
id: d64d6a8d-eb8e-11eb-b029-96fd216dec3c
last_name: Súkeník
- first_name: Christoph
full_name: Lampert, Christoph
id: 40C20FD2-F248-11E8-B48F-1D18A9856A87
last_name: Lampert
orcid: 0000-0001-8622-7887
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: 'Súkeník P, Lampert C, Mondelli M. Neural collapse is globally optimal in deep
regularized ResNets and transformers. In: 39th Conference on Neural Information
Processing Systems. Vol 38. Neural Information Processing Systems Foundation;
2025:48646-48677. doi:10.52202/085713-1450'
apa: 'Súkeník, P., Lampert, C., & Mondelli, M. (2025). Neural collapse is globally
optimal in deep regularized ResNets and transformers. In 39th Conference on
Neural Information Processing Systems (Vol. 38, pp. 48646–48677). San Diego,
CA, United States: Neural Information Processing Systems Foundation. https://doi.org/10.52202/085713-1450'
chicago: Súkeník, Peter, Christoph Lampert, and Marco Mondelli. “Neural Collapse
Is Globally Optimal in Deep Regularized ResNets and Transformers.” In 39th
Conference on Neural Information Processing Systems, 38:48646–77. Neural Information
Processing Systems Foundation, 2025. https://doi.org/10.52202/085713-1450.
ieee: P. Súkeník, C. Lampert, and M. Mondelli, “Neural collapse is globally optimal
in deep regularized ResNets and transformers,” in 39th Conference on Neural
Information Processing Systems, San Diego, CA, United States, 2025, vol. 38,
pp. 48646–48677.
ista: 'Súkeník P, Lampert C, Mondelli M. 2025. Neural collapse is globally optimal
in deep regularized ResNets and transformers. 39th Conference on Neural Information
Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in
Neural Information Processing Systems, vol. 38, 48646–48677.'
mla: Súkeník, Peter, et al. “Neural Collapse Is Globally Optimal in Deep Regularized
ResNets and Transformers.” 39th Conference on Neural Information Processing
Systems, vol. 38, Neural Information Processing Systems Foundation, 2025,
pp. 48646–77, doi:10.52202/085713-1450.
short: P. Súkeník, C. Lampert, M. Mondelli, in:, 39th Conference on Neural Information
Processing Systems, Neural Information Processing Systems Foundation, 2025, pp.
48646–48677.
conference:
end_date: 2025-12-07
location: San Diego, CA, United States
name: 'NeurIPS: Neural Information Processing Systems'
start_date: 2025-12-02
corr_author: '1'
das_tickbox: '0'
date_created: 2026-09-06T22:01:59Z
date_published: 2025-12-02T00:00:00Z
date_updated: 2026-09-10T08:11:46Z
day: '02'
ddc:
- '000'
department:
- _id: MaMo
- _id: GradSch
- _id: ChLa
doi: 10.52202/085713-1450
external_id:
arxiv:
- '2505.15239'
fulldoi: https://doi.org/10.52202/085713-1450
intvolume: ' 38'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.52202/085713-1450
month: '12'
oa: 1
oa_version: Published Version
page: 48646-48677
project:
- _id: 911e6d1f-16d5-11f0-9cad-c5c68c6a1cdf
grant_number: '101161364'
name: 'Inference in High Dimensions: Light-speed Algorithms and Information Limits'
publication: 39th Conference on Neural Information Processing Systems
publication_identifier:
isbn:
- '9798331338275'
issn:
- 1049-5258
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
researchdata_availability: no
scopus_import: '1'
status: public
supplementarymaterial: no
title: Neural collapse is globally optimal in deep regularized ResNets and transformers
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 38
year: '2025'
...
---
OA_place: publisher
OA_type: gold
_id: '21324'
abstract:
- lang: eng
text: Learning models have been shown to rely on spurious correlations between non-predictive
features and the associated labels in the training data, with negative implications
on robustness, bias and fairness. In this work, we provide a statistical characterization
of this phenomenon for high-dimensional regression, when the data contains a predictive
core feature x and a spurious feature y. Specifically, we quantify the amount
of spurious correlations C learned via linear regression, in terms of the data
covariance and the strength λ of the ridge regularization. As a consequence, we
first capture the simplicity of y through the spectrum of its covariance, and
its correlation with x through the Schur complement of the full data covariance.
Next, we prove a trade-off between C and the in-distribution test loss L, by showing
that the value of λ that minimizes L lies in an interval where C is increasing.
Finally, we investigate the effects of over-parameterization via the random features
model, by showing its equivalence to regularized linear regression. Our theoretical
results are supported by numerical experiments on Gaussian, Color-MNIST, and CIFAR-10
datasets.
acknowledgement: Marco Mondelli is funded by the European Union (ERC, INF2, project
number 101161364). Views and opinions expressed are however those of the author(s)
only and do not necessarily reflect those of the European Union or the European
Research Council Executive Agency. Neither the European Union nor the granting authority
can be held responsible for them. Simone Bombari is supported by a Google PhD fellowship.
The authors would like to thank GuanWen Qiu for helpful discussions.
alternative_title:
- PMLR
article_processing_charge: No
arxiv: 1
author:
- first_name: Simone
full_name: Bombari, Simone
id: ca726dda-de17-11ea-bc14-f9da834f63aa
last_name: Bombari
- first_name: Marco
full_name: Mondelli, Marco
id: 27EB676C-8706-11E9-9510-7717E6697425
last_name: Mondelli
orcid: 0000-0002-3242-7020
citation:
ama: 'Bombari S, Mondelli M. Spurious correlations in high dimensional regression:
The roles of regularization, simplicity bias and over-parameterization. In: Proceedings
of the 42nd International Conference on Machine Learning. Vol 267. ML Research
Press; 2025:4839-4873.'
apa: 'Bombari, S., & Mondelli, M. (2025). Spurious correlations in high dimensional
regression: The roles of regularization, simplicity bias and over-parameterization.
In Proceedings of the 42nd International Conference on Machine Learning
(Vol. 267, pp. 4839–4873). Vancouver, Canada: ML Research Press.'
chicago: 'Bombari, Simone, and Marco Mondelli. “Spurious Correlations in High Dimensional
Regression: The Roles of Regularization, Simplicity Bias and over-Parameterization.”
In Proceedings of the 42nd International Conference on Machine Learning,
267:4839–73. ML Research Press, 2025.'
ieee: 'S. Bombari and M. Mondelli, “Spurious correlations in high dimensional regression:
The roles of regularization, simplicity bias and over-parameterization,” in Proceedings
of the 42nd International Conference on Machine Learning, Vancouver, Canada,
2025, vol. 267, pp. 4839–4873.'
ista: 'Bombari S, Mondelli M. 2025. Spurious correlations in high dimensional regression:
The roles of regularization, simplicity bias and over-parameterization. Proceedings
of the 42nd International Conference on Machine Learning. ICML: International
Conference on Machine Learning, PMLR, vol. 267, 4839–4873.'
mla: 'Bombari, Simone, and Marco Mondelli. “Spurious Correlations in High Dimensional
Regression: The Roles of Regularization, Simplicity Bias and over-Parameterization.”
Proceedings of the 42nd International Conference on Machine Learning, vol.
267, ML Research Press, 2025, pp. 4839–73.'
short: S. Bombari, M. Mondelli, in:, Proceedings of the 42nd International Conference
on Machine Learning, ML Research Press, 2025, pp. 4839–4873.
conference:
end_date: 2025-07-19
location: Vancouver, Canada
name: 'ICML: International Conference on Machine Learning'
start_date: 2025-07-13
corr_author: '1'
date_created: 2026-02-18T11:58:00Z
date_published: 2025-07-30T00:00:00Z
date_updated: 2026-09-21T13:07:01Z
day: '30'
ddc:
- '000'
department:
- _id: MaMo
external_id:
arxiv:
- '2502.01347'
file:
- access_level: open_access
checksum: d4ba4f7717b362ca38878f45e57bd643
content_type: application/pdf
creator: dernst
date_created: 2026-02-19T08:04:38Z
date_updated: 2026-02-19T08:04:38Z
file_id: '21335'
file_name: 2025_ICML_Bombari.pdf
file_size: 887526
relation: main_file
success: 1
file_date_updated: 2026-02-19T08:04:38Z
has_accepted_license: '1'
intvolume: ' 267'
language:
- iso: eng
month: '07'
oa: 1
oa_version: Published Version
page: 4839-4873
project:
- _id: 911e6d1f-16d5-11f0-9cad-c5c68c6a1cdf
grant_number: '101161364'
name: 'Inference in High Dimensions: Light-speed Algorithms and Information Limits'
- _id: 92099302-16d5-11f0-9cad-f9a785f54fbd
name: 'Trustworthy Deep Learning Theory: Private Over-Parameterized Models and Robust
LLMs'
publication: Proceedings of the 42nd International Conference on Machine Learning
publication_identifier:
eissn:
- 2640-3498
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
related_material:
record:
- id: '22857'
relation: dissertation_contains
status: public
status: public
title: 'Spurious correlations in high dimensional regression: The roles of regularization,
simplicity bias and over-parameterization'
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 267
year: '2025'
...