{"volume":235,"type":"conference","related_material":{"record":[{"relation":"dissertation_contains","status":"for_moderation","id":"23072"},{"relation":"dissertation_contains","status":"public","id":"17465"}]},"status":"public","date_updated":"2026-10-10T22:31:20Z","_id":"17469","date_published":"2024-07-01T00:00:00Z","abstract":[{"text":"Autoencoders are a prominent model in many empirical branches of machine learning and lossy data compression. However, basic theoretical questions remain unanswered even in a shallow two-layer setting. In particular, to what degree does a shallow autoencoder capture the structure of the underlying data distribution? For the prototypical case of the 1-bit compression of sparse Gaussian data, we prove that gradient descent converges to a solution that completely disregards the sparse structure of the input. Namely, the performance of the algorithm is the same as if it was compressing a Gaussian source - with no sparsity. For general data distributions, we give evidence of a phase transition phenomenon in the shape of the gradient descent minimizer, as a function of the data sparsity: below the critical sparsity level, the minimizer is a rotation taken uniformly at random (just like in the compression of non-sparse data); above the critical sparsity, the minimizer is the identity (up to a permutation). Finally, by exploiting a connection with approximate message passing algorithms, we show how to improve upon Gaussian performance for the compression of sparse data: adding a denoising function to a shallow architecture already reduces the loss provably, and a suitable multi-layer decoder leads to a further improvement. We validate our findings on image datasets, such as CIFAR-10 and MNIST.","lang":"eng"}],"external_id":{"arxiv":["2402.05013"]},"title":"Compression of structured data with autoencoders: Provable benefit of nonlinearities and depth","oa":1,"year":"2024","main_file_link":[{"open_access":"1","url":"https://proceedings.mlr.press/v235/kogler24a.html"}],"arxiv":1,"citation":{"ieee":"K. Kögler, A. Shevchenko, H. Hassani, and M. Mondelli, “Compression of structured data with autoencoders: Provable benefit of nonlinearities and depth,” in <i>Proceedings of the 41st International Conference on Machine Learning</i>, Vienna, Austria, 2024, vol. 235, pp. 24964–25015.","ista":"Kögler K, Shevchenko A, Hassani H, Mondelli M. 2024. Compression of structured data with autoencoders: Provable benefit of nonlinearities and depth. Proceedings of the 41st International Conference on Machine Learning. ICML: International Conference on Machine Learning, PMLR, vol. 235, 24964–25015.","apa":"Kögler, K., Shevchenko, A., Hassani, H., &#38; Mondelli, M. (2024). Compression of structured data with autoencoders: Provable benefit of nonlinearities and depth. In <i>Proceedings of the 41st International Conference on Machine Learning</i> (Vol. 235, pp. 24964–25015). Vienna, Austria: ML Research Press.","short":"K. Kögler, A. Shevchenko, H. Hassani, M. Mondelli, in:, Proceedings of the 41st International Conference on Machine Learning, ML Research Press, 2024, pp. 24964–25015.","chicago":"Kögler, Kevin, Alexander Shevchenko, Hamed Hassani, and Marco Mondelli. “Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth.” In <i>Proceedings of the 41st International Conference on Machine Learning</i>, 235:24964–15. ML Research Press, 2024.","ama":"Kögler K, Shevchenko A, Hassani H, Mondelli M. Compression of structured data with autoencoders: Provable benefit of nonlinearities and depth. In: <i>Proceedings of the 41st International Conference on Machine Learning</i>. Vol 235. ML Research Press; 2024:24964-25015.","mla":"Kögler, Kevin, et al. “Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth.” <i>Proceedings of the 41st International Conference on Machine Learning</i>, vol. 235, ML Research Press, 2024, pp. 24964–5015."},"researchdata_availability":"no","intvolume":"       235","das_tickbox":"0","alternative_title":["PMLR"],"publication":"Proceedings of the 41st International Conference on Machine Learning","ddc":["000"],"language":[{"iso":"eng"}],"publication_status":"published","department":[{"_id":"DaAl"},{"_id":"MaMo"}],"day":"01","conference":{"end_date":"2024-07-27","location":"Vienna, Austria","start_date":"2024-07-21","name":"ICML: International Conference on Machine Learning"},"publisher":"ML Research Press","scopus_import":"1","acknowledgement":"Kevin Kogler, Alexander Shevchenko and Marco Mondelli are supported by the 2019 Lopez-Loreta Prize. Hamed\r\nHassani acknowledges the support by the NSF CIF award (1910056) and the NSF Institute for CORE Emerging Methods in Data Science (EnCORE).","project":[{"name":"Prix Lopez-Loretta 2019 - Marco Mondelli","_id":"059876FA-7A3F-11EA-A408-12923DDC885E"}],"author":[{"first_name":"Kevin","full_name":"Kögler, Kevin","last_name":"Kögler","id":"94ec913c-dc85-11ea-9058-e5051ab2428b"},{"id":"F2B06EC2-C99E-11E9-89F0-752EE6697425","last_name":"Shevchenko","full_name":"Shevchenko, Aleksandr","first_name":"Aleksandr"},{"full_name":"Hassani, Hamed","first_name":"Hamed","last_name":"Hassani"},{"last_name":"Mondelli","orcid":"0000-0002-3242-7020","id":"27EB676C-8706-11E9-9510-7717E6697425","full_name":"Mondelli, Marco","first_name":"Marco"}],"date_created":"2024-08-29T11:47:57Z","page":"24964-25015","corr_author":"1","user_id":"317138e5-6ab7-11ef-aa6d-ffef3953e345","supplementarymaterial":"no","quality_controlled":"1","month":"07","article_processing_charge":"No","oa_version":"Published Version"}