{"abstract":[{"lang":"eng","text":"Deep neural networks (DNNs) at convergence consistently represent the training data in the last layer via a geometric structure referred to as neural collapse. This empirical evidence has spurred a line of theoretical research aimed at proving the emergence of neural collapse, mostly focusing on the unconstrained features model. Here, the features of the penultimate layer are free variables, which makes the model data-agnostic and puts into question its ability to capture DNN training. Our work addresses the issue, moving away from unconstrained features and\r\nstudying DNNs that end with at least two linear layers. We first prove generic guarantees on neural collapse that assume (i) low training error and balancedness of linear layers (for within-class variability collapse), and (ii) bounded conditioning of the features before the linear part (for orthogonality of class-means, and their alignment with weight matrices). The balancedness refers to the fact that W⊤ℓ+1Wℓ+1 ≈ WℓW⊤ℓfor any pair of consecutive weight matrices of the linear part, and the bounded conditioning requires a well-behaved ratio between largest and smallest non-zero singular values of the features. We then show that such assumptions hold for gradient descent training with weight decay: (i) for networks with a wide first layer, we prove low training error and balancedness, and (ii) for solutions that are either nearly optimal or stable under large learning rates, we additionally prove the bounded conditioning. Taken together, our results are the first to show neural collapse in the end-to-end training of DNNs."}],"project":[{"_id":"911e6d1f-16d5-11f0-9cad-c5c68c6a1cdf","grant_number":"101161364","name":"Inference in High Dimensions: Light-speed Algorithms and Information Limits"}],"status":"public","language":[{"iso":"eng"}],"OA_place":"publisher","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","external_id":{"arxiv":["2410.04887"]},"OA_type":"diamond","author":[{"first_name":"Arthur","last_name":"Jacot","full_name":"Jacot, Arthur"},{"first_name":"Peter","full_name":"Súkeník, Peter","last_name":"Súkeník","id":"d64d6a8d-eb8e-11eb-b029-96fd216dec3c"},{"first_name":"Zihan","last_name":"Wang","full_name":"Wang, Zihan"},{"orcid":"0000-0002-3242-7020","full_name":"Mondelli, Marco","last_name":"Mondelli","first_name":"Marco","id":"27EB676C-8706-11E9-9510-7717E6697425"}],"citation":{"short":"A. Jacot, P. Súkeník, Z. Wang, M. Mondelli, in:, 13th International Conference on Learning Representations, ICLR, 2025, pp. 1905–1931.","ama":"Jacot A, Súkeník P, Wang Z, Mondelli M. Wide neural networks trained with weight decay provably exhibit neural collapse. In: <i>13th International Conference on Learning Representations</i>. ICLR; 2025:1905-1931.","apa":"Jacot, A., Súkeník, P., Wang, Z., &#38; Mondelli, M. (2025). Wide neural networks trained with weight decay provably exhibit neural collapse. In <i>13th International Conference on Learning Representations</i> (pp. 1905–1931). Singapore, Singapore: ICLR.","chicago":"Jacot, Arthur, Peter Súkeník, Zihan Wang, and Marco Mondelli. “Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse.” In <i>13th International Conference on Learning Representations</i>, 1905–31. ICLR, 2025.","mla":"Jacot, Arthur, et al. “Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse.” <i>13th International Conference on Learning Representations</i>, ICLR, 2025, pp. 1905–31.","ieee":"A. Jacot, P. Súkeník, Z. Wang, and M. Mondelli, “Wide neural networks trained with weight decay provably exhibit neural collapse,” in <i>13th International Conference on Learning Representations</i>, Singapore, Singapore, 2025, pp. 1905–1931.","ista":"Jacot A, Súkeník P, Wang Z, Mondelli M. 2025. Wide neural networks trained with weight decay provably exhibit neural collapse. 13th International Conference on Learning Representations. ICLR: International Conference on Learning Representations, 1905–1931."},"tmp":{"short":"CC BY (4.0)","legal_code_url":"https://creativecommons.org/licenses/by/4.0/legalcode","name":"Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)","image":"/images/cc_by.png"},"department":[{"_id":"MaMo"}],"article_processing_charge":"No","oa":1,"acknowledgement":"M. M. and P. S. are funded by the European Union (ERC, INF2, project number 101161364). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.","_id":"20035","month":"04","year":"2025","arxiv":1,"file":[{"access_level":"open_access","file_name":"2025_ICLR_Jacot.pdf","file_id":"20114","relation":"main_file","date_updated":"2025-08-04T08:45:43Z","date_created":"2025-08-04T08:45:43Z","success":1,"file_size":1337236,"checksum":"59c48c173887139647cc9839c0801136","creator":"dernst","content_type":"application/pdf"}],"title":"Wide neural networks trained with weight decay provably exhibit neural collapse","has_accepted_license":"1","publisher":"ICLR","publication_status":"published","scopus_import":"1","conference":{"location":"Singapore, Singapore","end_date":"2025-04-28","start_date":"2025-04-24","name":"ICLR: International Conference on Learning Representations"},"quality_controlled":"1","oa_version":"Published Version","file_date_updated":"2025-08-04T08:45:43Z","corr_author":"1","publication":"13th International Conference on Learning Representations","day":"01","date_updated":"2025-08-04T08:47:00Z","date_created":"2025-07-20T22:02:02Z","type":"conference","date_published":"2025-04-01T00:00:00Z","ddc":["000"],"page":"1905-1931","publication_identifier":{"isbn":["9798331320850"]}}