conference paper published yes Peter Súkeník author d64d6a8d-eb8e-11eb-b029-96fd216dec3c Christoph Lampert author 40C20FD2-F248-11E8-B48F-1D18A9856A870000-0001-8622-7887 Marco Mondelli author 27EB676C-8706-11E9-9510-7717E66974250000-0002-3242-7020 MaMo department GradSch department ChLa department NeurIPS: Neural Information Processing Systems Inference in High Dimensions: Light-speed Algorithms and Information Limits project The empirical emergence of neural collapse—a surprising symmetry in the feature representations of the training data in the penultimate layer of deep neural networks—has spurred a line of theoretical research aimed at its understanding. However, existing work focuses on data-agnostic models or, when data structure is taken into account, it remains limited to multi-layer perceptrons. Our paper fills both these gaps by analyzing modern architectures in a data-aware regime: we prove that global optima of deep regularized transformers and residual networks (ResNets) with LayerNorm trained with cross entropy or mean squared error loss are approximately collapsed, and the approximation gets tighter as the depth grows. More generally, we formally reduce any end-to-end large-depth ResNet or transformer training into an equivalent unconstrained features model, thus justifying its wide use in the literature even beyond data-agnostic settings. Our theoretical results are supported by experiments on computer vision and language datasets showing that, as the depth grows, neural collapse indeed becomes more prominent. Neural Information Processing Systems Foundation2025San Diego, CA, United States eng 1049-5258 9798331338275 2505.1523910.52202/085713-1450 3848646-48677 Súkeník, P., Lampert, C., &#38; Mondelli, M. (2025). Neural collapse is globally optimal in deep regularized ResNets and transformers. In <i>39th Conference on Neural Information Processing Systems</i> (Vol. 38, pp. 48646–48677). San Diego, CA, United States: Neural Information Processing Systems Foundation. <a href="https://doi.org/10.52202/085713-1450">https://doi.org/10.52202/085713-1450</a> P. Súkeník, C. Lampert, M. Mondelli, in:, 39th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2025, pp. 48646–48677. Súkeník, Peter, Christoph Lampert, and Marco Mondelli. “Neural Collapse Is Globally Optimal in Deep Regularized ResNets and Transformers.” In <i>39th Conference on Neural Information Processing Systems</i>, 38:48646–77. Neural Information Processing Systems Foundation, 2025. <a href="https://doi.org/10.52202/085713-1450">https://doi.org/10.52202/085713-1450</a>. P. Súkeník, C. Lampert, and M. Mondelli, “Neural collapse is globally optimal in deep regularized ResNets and transformers,” in <i>39th Conference on Neural Information Processing Systems</i>, San Diego, CA, United States, 2025, vol. 38, pp. 48646–48677. Súkeník P, Lampert C, Mondelli M. Neural collapse is globally optimal in deep regularized ResNets and transformers. In: <i>39th Conference on Neural Information Processing Systems</i>. Vol 38. Neural Information Processing Systems Foundation; 2025:48646-48677. doi:<a href="https://doi.org/10.52202/085713-1450">10.52202/085713-1450</a> Súkeník, Peter, et al. “Neural Collapse Is Globally Optimal in Deep Regularized ResNets and Transformers.” <i>39th Conference on Neural Information Processing Systems</i>, vol. 38, Neural Information Processing Systems Foundation, 2025, pp. 48646–77, doi:<a href="https://doi.org/10.52202/085713-1450">10.52202/085713-1450</a>. Súkeník P, Lampert C, Mondelli M. 2025. Neural collapse is globally optimal in deep regularized ResNets and transformers. 39th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 38, 48646–48677. 228252026-09-06T22:01:59Z2026-09-10T08:11:46Z