--- OA_place: repository OA_type: green _id: '22827' abstract: - lang: eng text: Quantized training of Large Language Models (LLMs) remains an open challenge, as maintaining accuracy while performing all matrix multiplications in low precision has proven difficult. This is particularly the case when fine-tuning pre-trained models, which can have large weight, activation, and error (output gradient) outlier values that make lower-precision optimization difficult. To address this, we present HALO, a new quantization-aware training approach for Transformers that enables accurate and efficient low-precision training by combining 1) strategic placement of Hadamard rotations in both forward and backward passes, which mitigate outliers, 2) high-performance kernel support, and 3) FSDP integration for low-precision communication. Our approach ensures that all large matrix multiplications during the forward and backward passes are executed in lower precision. Applied to LLaMa models, HALO achieves near-full-precision-equivalent results during fine-tuning on various tasks, while delivering up to 1.41x end-to-end speedup for full fine-tuning on RTX 4090 GPUs. HALO efficiently supports both standard and parameter-efficient fine-tuning (PEFT). Our results demonstrate the first practical approach to fully quantized LLM fine-tuning that maintains accuracy in INT8 and FP6 precision, while delivering performance benefits. acknowledgement: "This project has received funding from the European Research Council (ERC) under the European\r\nUnion’s Horizon 2020 program (grant agreement PSAP, No. 101002047. This research also obtained\r\nfunding from the “UrbanTwin: An urban digital twin for climate action: Assessing policies and\r\nsolutions for energy, water and infrastructure” project, funded by the ETH-Domain Joint Initiative\r\nprogram in the Strategic Area Energy, Climate and Sustainable Environment." alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Saleh full_name: Ashkboos, Saleh last_name: Ashkboos - first_name: Mahdi full_name: Nikdan, Mahdi id: 66374281-f394-11eb-9cf6-869147deecc0 last_name: Nikdan - first_name: Soroush full_name: Tabesh, Soroush id: 06000900-6068-11ef-8d61-c2472ef2e752 last_name: Tabesh orcid: 0009-0003-4119-6281 - first_name: Roberto full_name: Lopez Castro, Roberto id: 18495c31-fb57-11ef-ba0d-c290e10e394e last_name: Lopez Castro - first_name: Torsten full_name: Hoefler, Torsten last_name: Hoefler - first_name: Dan-Adrian full_name: Alistarh, Dan-Adrian id: 4A899BFC-F248-11E8-B48F-1D18A9856A87 last_name: Alistarh orcid: 0000-0003-3650-940X citation: ama: 'Ashkboos S, Nikdan M, Tabesh S, Lopez Castro R, Hoefler T, Alistarh D-A. HALO: Hadamard-assisted lower-precision optimization for LLMs. In: 39th Conference on Neural Information Processing Systems. Vol 38. Neural Information Processing Systems Foundation; 2025:131755-131780. doi:10.52202/085713-3966' apa: 'Ashkboos, S., Nikdan, M., Tabesh, S., Lopez Castro, R., Hoefler, T., & Alistarh, D.-A. (2025). HALO: Hadamard-assisted lower-precision optimization for LLMs. In 39th Conference on Neural Information Processing Systems (Vol. 38, pp. 131755–131780). San Diego, CA, United States: Neural Information Processing Systems Foundation. https://doi.org/10.52202/085713-3966' chicago: 'Ashkboos, Saleh, Mahdi Nikdan, Soroush Tabesh, Roberto Lopez Castro, Torsten Hoefler, and Dan-Adrian Alistarh. “HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs.” In 39th Conference on Neural Information Processing Systems, 38:131755–80. Neural Information Processing Systems Foundation, 2025. https://doi.org/10.52202/085713-3966.' ieee: 'S. Ashkboos, M. Nikdan, S. Tabesh, R. Lopez Castro, T. Hoefler, and D.-A. Alistarh, “HALO: Hadamard-assisted lower-precision optimization for LLMs,” in 39th Conference on Neural Information Processing Systems, San Diego, CA, United States, 2025, vol. 38, pp. 131755–131780.' ista: 'Ashkboos S, Nikdan M, Tabesh S, Lopez Castro R, Hoefler T, Alistarh D-A. 2025. HALO: Hadamard-assisted lower-precision optimization for LLMs. 39th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 38, 131755–131780.' mla: 'Ashkboos, Saleh, et al. “HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs.” 39th Conference on Neural Information Processing Systems, vol. 38, Neural Information Processing Systems Foundation, 2025, pp. 131755–80, doi:10.52202/085713-3966.' short: S. Ashkboos, M. Nikdan, S. Tabesh, R. Lopez Castro, T. Hoefler, D.-A. Alistarh, in:, 39th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2025, pp. 131755–131780. conference: end_date: 2025-12-07 location: San Diego, CA, United States name: 'NeurIPS: Neural Information Processing Systems' start_date: 2025-12-02 das_tickbox: '0' date_created: 2026-09-06T22:02:00Z date_published: 2025-12-01T00:00:00Z date_updated: 2026-09-17T06:39:18Z day: '01' department: - _id: DaAl - _id: GradSch doi: 10.52202/085713-3966 external_id: arxiv: - '2501.02625' fulldoi: https://doi.org/10.52202/085713-3966 intvolume: ' 38' language: - iso: eng main_file_link: - open_access: '1' url: https://doi.org/10.48550/arXiv.2501.02625 month: '12' oa: 1 oa_version: Preprint page: 131755-131780 publication: 39th Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 isbn: - '9798331338275' publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' researchdata_availability: no scopus_import: '1' status: public supplementarymaterial: no title: 'HALO: Hadamard-assisted lower-precision optimization for LLMs' type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 38 year: '2025' ... --- _id: '22831' abstract: - lang: eng text: "Neural models learn representations of high-dimensional data on low-dimensional\r\nmanifolds. Multiple factors, including stochasticities in the training process, model\r\narchitectures, and additional inductive biases, may induce different representations,\r\neven when learning the same task on the same data. However, it has recently been\r\nshown that when a latent structure is shared between distinct latent spaces, relative\r\ndistances between representations can be preserved, up to distortions. Building\r\non this idea, we demonstrate that exploiting the differential-geometric structure of\r\nlatent spaces of neural models, it is possible to capture precisely the transformations\r\nbetween representational spaces trained on similar data distributions. Specifically,\r\nwe assume that distinct neural models parametrize approximately the same underlying manifold, and introduce a representation based on the pullback metric\r\nthat captures the intrinsic structure of the latent space, while scaling efficiently\r\nto large models. We validate experimentally our method on model stitching and\r\nretrieval tasks, covering autoencoders and vision foundation discriminative models,\r\nacross diverse architectures, datasets, pretraining schemes and modalities. Code is\r\navailable at https://github.com/marc0git/RelativeGeodesics." acknowledgement: "We thank Gregor Krzmanc, German Magai, Vital Fernandez for insightful discussions in the early\r\nstages of the project. HY was supported by the Research Council of Finland Flagship programme:\r\nFinnish Center for Artificial Intelligence FCAI. HY wishes to acknowledge CSC - IT Center for\r\nScience, Finland, for computational resources. GA was supported by the DFF Sapere Aude Starting\r\nGrant “GADL”. SH was supported by a research grant (42062) from VILLUM FONDEN and partly\r\nfunded by the Novo Nordisk Foundation through the Center for Basic Research in Life Science\r\n(NNF20OC0062606). SH received funding from the European Research Council (ERC) under the\r\nEuropean Union’s Horizon Programme (grant agreement 101125003). MF is supported by the MSCA\r\nIST-Bridge fellowship which has received funding from the European Union’s Horizon 2020 research\r\nand innovation program under the Marie Skłodowska-Curie grant agreement No 101034413." alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Hanlin full_name: Yu, Hanlin last_name: Yu - first_name: Berfin full_name: Inal, Berfin last_name: Inal - first_name: Georgios full_name: Arvanitidis, Georgios last_name: Arvanitidis - first_name: Søren full_name: Hauberg, Søren last_name: Hauberg - first_name: Francesco full_name: Locatello, Francesco id: 26cfd52f-2483-11ee-8040-88983bcc06d4 last_name: Locatello orcid: 0000-0002-4850-0683 - first_name: Marco full_name: Fumero, Marco id: 1c1593eb-393f-11ef-bb8e-ab4f1e979650 last_name: Fumero citation: ama: 'Yu H, Inal B, Arvanitidis G, Hauberg S, Locatello F, Fumero M. Connecting neural models latent geometries with relative geodesic representations. In: 39th Conference on Neural Information Processing Systems. Vol 38. Neural Information Processing Systems Foundation; 2025:125316-125360. doi:10.52202/085713-3769' apa: 'Yu, H., Inal, B., Arvanitidis, G., Hauberg, S., Locatello, F., & Fumero, M. (2025). Connecting neural models latent geometries with relative geodesic representations. In 39th Conference on Neural Information Processing Systems (Vol. 38, pp. 125316–125360). San Diego, CA, United States: Neural Information Processing Systems Foundation. https://doi.org/10.52202/085713-3769' chicago: Yu, Hanlin, Berfin Inal, Georgios Arvanitidis, Søren Hauberg, Francesco Locatello, and Marco Fumero. “Connecting Neural Models Latent Geometries with Relative Geodesic Representations.” In 39th Conference on Neural Information Processing Systems, 38:125316–60. Neural Information Processing Systems Foundation, 2025. https://doi.org/10.52202/085713-3769. ieee: H. Yu, B. Inal, G. Arvanitidis, S. Hauberg, F. Locatello, and M. Fumero, “Connecting neural models latent geometries with relative geodesic representations,” in 39th Conference on Neural Information Processing Systems, San Diego, CA, United States, 2025, vol. 38, pp. 125316–125360. ista: 'Yu H, Inal B, Arvanitidis G, Hauberg S, Locatello F, Fumero M. 2025. Connecting neural models latent geometries with relative geodesic representations. 39th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 38, 125316–125360.' mla: Yu, Hanlin, et al. “Connecting Neural Models Latent Geometries with Relative Geodesic Representations.” 39th Conference on Neural Information Processing Systems, vol. 38, Neural Information Processing Systems Foundation, 2025, pp. 125316–60, doi:10.52202/085713-3769. short: H. Yu, B. Inal, G. Arvanitidis, S. Hauberg, F. Locatello, M. Fumero, in:, 39th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2025, pp. 125316–125360. conference: end_date: 2025-12-07 location: San Diego, CA, United States name: 'NeurIPS: Neural Information Processing Systems' start_date: 2025-12-02 corr_author: '1' das_tickbox: '0' date_created: 2026-09-06T22:02:01Z date_published: 2025-12-01T00:00:00Z date_updated: 2026-09-17T07:09:28Z day: '01' department: - _id: FrLo doi: 10.52202/085713-3769 ec_funded: 1 external_id: arxiv: - '2506.01599' fulldoi: https://doi.org/10.52202/085713-3769 intvolume: ' 38' language: - iso: eng month: '12' oa_version: None page: 125316-125360 project: - _id: fc2ed2f7-9c52-11eb-aca3-c01059dda49c call_identifier: H2020 grant_number: '101034413' name: 'IST-BRIDGE: International postdoctoral program' publication: 39th Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 isbn: - '9798331338275' publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' related_material: link: - relation: software url: ' https://github.com/marc0git/RelativeGeodesics' researchdata_availability: yes scopus_import: '1' status: public supplementarymaterial: yes title: Connecting neural models latent geometries with relative geodesic representations type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 38 year: '2025' ... --- OA_place: publisher OA_type: free access _id: '21076' abstract: - lang: eng text: In many scientific experiments, the data annotating cost constraints the pace for testing novel hypotheses. Yet, modern machine learning pipelines offer a promising solution—provided their predictions yield correct conclusions. We focus on Prediction-Powered Causal Inferences (PPCI), i.e., estimating the treatment effect in an unlabeled target experiment, relying on training data with the same outcome annotated but potentially different treatment or effect modifiers. We first show that conditional calibration guarantees valid PPCI at population level. Then, we introduce a sufficient representation constraint transferring validity across experiments, which we propose to enforce in practice in Deconfounded Empirical Risk Minimization, our new model-agnostic training objective. We validate our method on synthetic and real-world scientific data, solving impossible problem instances for Empirical Risk Minimization even with standard invariance constraints. In particular, for the first time, we achieve valid causal inference on a scientific experiment with complex recording and no human annotations, fine-tuning a foundational model on our similar annotated experiment. acknowledgement: We thank the Causal Learning and Artificial Intelligence group at ISTA for the continuous feedback on the project and valuable discussions. We thank the Social Immunity group at ISTA, particularly Jinook Oh, for the annotation program and Michaela Hoenigsberger for supporting our ecological experiment. Riccardo Cadei is supported by a Google Research Scholar Award and a Google Initiated Gift to Francesco Locatello. This research was funded in part by the Austrian Science Fund (FWF) 10.55776/COE12). It was further partially supported by the ISTA Interdisciplinary Project Committee for the collaborative project “ALED” between Francesco Locatello and Sylvia Cremer. For open access purposes, the author has applied a CC BY public copyright license to any author accepted manuscript version arising from this submission. alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No author: - first_name: Riccardo full_name: Cadei, Riccardo id: 0fa8b76f-72f0-11ef-b75a-a5da96e5ad6b last_name: Cadei - first_name: Ilker full_name: Demirel, Ilker last_name: Demirel - first_name: Piersilvio full_name: De Bartolomeis, Piersilvio last_name: De Bartolomeis - first_name: Lukas full_name: Lindorfer, Lukas id: 85f0e6d3-06b3-11ec-8982-8c5049fa4455 last_name: Lindorfer - first_name: Sylvia full_name: Cremer, Sylvia id: 2F64EC8C-F248-11E8-B48F-1D18A9856A87 last_name: Cremer orcid: 0000-0002-2193-3868 - first_name: Cordelia full_name: Schmid, Cordelia last_name: Schmid - first_name: Francesco full_name: Locatello, Francesco id: 26cfd52f-2483-11ee-8040-88983bcc06d4 last_name: Locatello orcid: 0000-0002-4850-0683 citation: ama: 'Cadei R, Demirel I, De Bartolomeis P, et al. Prediction-powered causal inferences. In: 39th Annual Conference on Neural Information Processing Systems. Vol 38. Neural Information Processing Systems Foundation; 2025. doi:10.52202/085713-2479' apa: 'Cadei, R., Demirel, I., De Bartolomeis, P., Lindorfer, L., Cremer, S., Schmid, C., & Locatello, F. (2025). Prediction-powered causal inferences. In 39th Annual Conference on Neural Information Processing Systems (Vol. 38). San Diego, CA, United States: Neural Information Processing Systems Foundation. https://doi.org/10.52202/085713-2479' chicago: Cadei, Riccardo, Ilker Demirel, Piersilvio De Bartolomeis, Lukas Lindorfer, Sylvia Cremer, Cordelia Schmid, and Francesco Locatello. “Prediction-Powered Causal Inferences.” In 39th Annual Conference on Neural Information Processing Systems, Vol. 38. Neural Information Processing Systems Foundation, 2025. https://doi.org/10.52202/085713-2479. ieee: R. Cadei et al., “Prediction-powered causal inferences,” in 39th Annual Conference on Neural Information Processing Systems, San Diego, CA, United States, 2025, vol. 38. ista: 'Cadei R, Demirel I, De Bartolomeis P, Lindorfer L, Cremer S, Schmid C, Locatello F. 2025. Prediction-powered causal inferences. 39th Annual Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 38.' mla: Cadei, Riccardo, et al. “Prediction-Powered Causal Inferences.” 39th Annual Conference on Neural Information Processing Systems, vol. 38, Neural Information Processing Systems Foundation, 2025, doi:10.52202/085713-2479. short: R. Cadei, I. Demirel, P. De Bartolomeis, L. Lindorfer, S. Cremer, C. Schmid, F. Locatello, in:, 39th Annual Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2025. conference: end_date: 2025-12-07 location: San Diego, CA, United States name: 'NeurIPS: Neural Information Processing Systems' start_date: 2025-12-02 das_tickbox: '1' date_created: 2026-01-29T14:35:11Z date_published: 2025-12-15T00:00:00Z date_updated: 2026-10-15T09:29:52Z day: '15' ddc: - '000' department: - _id: FrLo - _id: SyCr doi: 10.52202/085713-2479 file: - access_level: open_access checksum: 92467fa566cd36671a6a3b9e71ae0f71 content_type: application/pdf creator: flocatel date_created: 2026-01-29T14:35:02Z date_updated: 2026-01-29T14:35:02Z file_id: '21077' file_name: 17546_Prediction_Powered_Causa.pdf file_size: 8489023 relation: main_file success: 1 file_date_updated: 2026-01-29T14:35:02Z fulldoi: https://doi.org/10.52202/085713-2479 has_accepted_license: '1' intvolume: ' 38' language: - iso: eng license: https://creativecommons.org/licenses/by/4.0/ month: '12' oa: 1 oa_version: Published Version project: - _id: a392d8f0-b034-11f1-b88e-d3025c6734f6 grant_number: COE12 name: Bilateral Artificial Intelligence (Locatello) publication: 39th Annual Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' status: public title: Prediction-powered causal inferences tmp: image: /images/cc_by.png legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0) short: CC BY (4.0) type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 38 year: '2025' ... --- OA_place: publisher OA_type: free access _id: '21072' abstract: - lang: eng text: Language and vision-language models have shown impressive performance across a wide range of tasks, but their internal mechanisms remain only partly understood. In this work, we study how individual attention heads in text-generative models specialize in specific semantic or visual attributes. Building on an established interpretability method, we reinterpret the practice of probing intermediate activations with the final decoding layer through the lens of signal processing. This lets us analyze multiple samples in a principled way and rank attention heads based on their relevance to target concepts. Our results show consistent patterns of specialization at the head level across both unimodal and multimodal transformers. Remarkably, we find that editing as few as 1% of the heads, selected using our method, can reliably suppress or enhance targeted concepts in the model output. We validate our approach on language tasks such as question answering and toxicity mitigation, as well as vision-language tasks including image classification and captioning. Our findings highlight an interpretable and controllable structure within attention layers, offering simple tools for understanding and editing large-scale generative models. acknowledgement: 'The authors acknowledge the Area Science Park supercomputing platform ORFEO made available for conducting the research reported in this paper, and the technical support of the Laboratory of Data Engineering staff. LB, DD and AC were supported by the project “Supporto alla diagnosi di malattie rare tramite l’intelligenza artificiale" CUP: F53C22001770002 and “Valutazione automatica delle immagini diagnostiche tramite l’intelligenza artificiale", CUP: F53C22001780002. LB was supported by the European Union – NextGenerationEU within the project PNRR “Finanziamento di progetti presentati da giovani ricercatori" - Mission 4 Component 2 Investment 1.2, CUP: J93C25000440001. AC was supported by the European Union – NextGenerationEU within the project PNRR “PRP@CERIC" IR0000028 - Mission 4 Component 2 Investment 3.1 Action 3.1.1. ' article_processing_charge: No arxiv: 1 author: - first_name: Lorenzo full_name: Basile, Lorenzo last_name: Basile - first_name: Valentino full_name: Maiorca, Valentino last_name: Maiorca - first_name: Diego full_name: Doimo, Diego last_name: Doimo - first_name: Francesco full_name: Locatello, Francesco id: 26cfd52f-2483-11ee-8040-88983bcc06d4 last_name: Locatello orcid: 0000-0002-4850-0683 - first_name: Alberto full_name: Cazzaniga, Alberto last_name: Cazzaniga citation: ama: 'Basile L, Maiorca V, Doimo D, Locatello F, Cazzaniga A. Head pursuit: Probing attention specialization in multimodal transformers. In: 39th Annual Conference on Neural Information Processing Systems. Vol 38. Neural Information Processing Systems Foundation; 2025. doi:10.52202/085713-3752' apa: 'Basile, L., Maiorca, V., Doimo, D., Locatello, F., & Cazzaniga, A. (2025). Head pursuit: Probing attention specialization in multimodal transformers. In 39th Annual Conference on Neural Information Processing Systems (Vol. 38). San Diego, CA, United States: Neural Information Processing Systems Foundation. https://doi.org/10.52202/085713-3752' chicago: 'Basile, Lorenzo, Valentino Maiorca, Diego Doimo, Francesco Locatello, and Alberto Cazzaniga. “Head Pursuit: Probing Attention Specialization in Multimodal Transformers.” In 39th Annual Conference on Neural Information Processing Systems, Vol. 38. Neural Information Processing Systems Foundation, 2025. https://doi.org/10.52202/085713-3752.' ieee: 'L. Basile, V. Maiorca, D. Doimo, F. Locatello, and A. Cazzaniga, “Head pursuit: Probing attention specialization in multimodal transformers,” in 39th Annual Conference on Neural Information Processing Systems, San Diego, CA, United States, 2025, vol. 38.' ista: 'Basile L, Maiorca V, Doimo D, Locatello F, Cazzaniga A. 2025. Head pursuit: Probing attention specialization in multimodal transformers. 39th Annual Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems vol. 38.' mla: 'Basile, Lorenzo, et al. “Head Pursuit: Probing Attention Specialization in Multimodal Transformers.” 39th Annual Conference on Neural Information Processing Systems, vol. 38, Neural Information Processing Systems Foundation, 2025, doi:10.52202/085713-3752.' short: L. Basile, V. Maiorca, D. Doimo, F. Locatello, A. Cazzaniga, in:, 39th Annual Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2025. conference: end_date: 2025-12-07 location: San Diego, CA, United States name: 'NeurIPS: Neural Information Processing Systems' start_date: 2025-12-02 das_tickbox: '1' date_created: 2026-01-29T14:29:23Z date_published: 2025-12-15T00:00:00Z date_updated: 2026-10-15T09:26:57Z day: '15' ddc: - '000' department: - _id: FrLo doi: 10.52202/085713-3752 external_id: arxiv: - '2510.21518' file: - access_level: open_access checksum: 85be3f98663e2595cf37001852b477cb content_type: application/pdf creator: flocatel date_created: 2026-01-29T14:29:14Z date_updated: 2026-01-29T14:29:14Z file_id: '21073' file_name: 2510.21518v2.pdf file_size: 4271547 relation: main_file success: 1 file_date_updated: 2026-01-29T14:29:14Z fulldoi: https://doi.org/10.52202/085713-3752 has_accepted_license: '1' intvolume: ' 38' language: - iso: eng month: '12' oa: 1 oa_version: Preprint publication: 39th Annual Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' status: public title: 'Head pursuit: Probing attention specialization in multimodal transformers' tmp: image: /images/cc_by.png legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0) short: CC BY (4.0) type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 38 year: '2025' ... --- OA_place: repository OA_type: green _id: '21068' abstract: - lang: eng text: "Causal reasoning and discovery, two fundamental tasks of causal analysis,\r\noften face challenges in applications due to the complexity, noisiness, and highdimensionality of real-world data. Despite recent progress in identifying latent\r\ncausal structures using causal representation learning (CRL), what makes learned\r\nrepresentations useful for causal downstream tasks and how to evaluate them are\r\nstill not well understood. In this paper, we reinterpret CRL using a measurement\r\nmodel framework, where the learned representations are viewed as proxy measurements of the latent causal variables. Our approach clarifies the conditions under\r\nwhich learned representations support downstream causal reasoning and provides\r\na principled basis for quantitatively assessing the quality of representations using\r\na new Test-based Measurement EXclusivity (T-MEX) score. We validate T-MEX\r\nacross diverse causal inference scenarios, including numerical simulations and\r\nreal-world ecological video analysis, demonstrating that the proposed framework\r\nand corresponding score effectively assess the identification of learned representations and their usefulness for causal downstream tasks. Reproducible code can\r\nbe found at https://github.com/shimenghuang/a-measurement-perspective-of-crl." acknowledgement: "This research was funded in whole or in part by the Austrian Science Fund (FWF) 10.55776/COE12. For open access purposes, the author has applied a CC BY public copyright license to any accepted manuscript version arising from this submission.\r\n" alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Dingling full_name: Yao, Dingling id: d3e02e50-48a8-11ee-8f62-c108061797fa last_name: Yao - first_name: Shimeng full_name: Huang, Shimeng id: 989c2a06-fb4e-11ef-a992-ab766442255b last_name: Huang orcid: 0000-0001-6919-821X - first_name: Riccardo full_name: Cadei, Riccardo id: 0fa8b76f-72f0-11ef-b75a-a5da96e5ad6b last_name: Cadei - first_name: Kun full_name: Zhang, Kun last_name: Zhang - first_name: Francesco full_name: Locatello, Francesco id: 26cfd52f-2483-11ee-8040-88983bcc06d4 last_name: Locatello orcid: 0000-0002-4850-0683 citation: ama: 'Yao D, Huang S, Cadei R, Zhang K, Locatello F. The third pillar of causal analysis? A measurement perspective on causal representations. In: 39th Annual Conference on Neural Information Processing Systems. Vol 38. Neural Information Processing Systems Foundation; 2025. doi:10.52202/085713-1444' apa: 'Yao, D., Huang, S., Cadei, R., Zhang, K., & Locatello, F. (2025). The third pillar of causal analysis? A measurement perspective on causal representations. In 39th Annual Conference on Neural Information Processing Systems (Vol. 38). San Diego, CA, United States: Neural Information Processing Systems Foundation. https://doi.org/10.52202/085713-1444' chicago: Yao, Dingling, Shimeng Huang, Riccardo Cadei, Kun Zhang, and Francesco Locatello. “The Third Pillar of Causal Analysis? A Measurement Perspective on Causal Representations.” In 39th Annual Conference on Neural Information Processing Systems, Vol. 38. Neural Information Processing Systems Foundation, 2025. https://doi.org/10.52202/085713-1444. ieee: D. Yao, S. Huang, R. Cadei, K. Zhang, and F. Locatello, “The third pillar of causal analysis? A measurement perspective on causal representations,” in 39th Annual Conference on Neural Information Processing Systems, San Diego, CA, United States, 2025, vol. 38. ista: 'Yao D, Huang S, Cadei R, Zhang K, Locatello F. 2025. The third pillar of causal analysis? A measurement perspective on causal representations. 39th Annual Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 38.' mla: Yao, Dingling, et al. “The Third Pillar of Causal Analysis? A Measurement Perspective on Causal Representations.” 39th Annual Conference on Neural Information Processing Systems, vol. 38, Neural Information Processing Systems Foundation, 2025, doi:10.52202/085713-1444. short: D. Yao, S. Huang, R. Cadei, K. Zhang, F. Locatello, in:, 39th Annual Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2025. conference: end_date: 2025-12-07 location: San Diego, CA, United States name: 'NeurIPS: Neural Information Processing Systems' start_date: 2025-12-02 corr_author: '1' das_tickbox: '1' date_created: 2026-01-29T14:24:56Z date_published: 2025-12-15T00:00:00Z date_updated: 2026-10-15T09:31:39Z day: '15' ddc: - '000' department: - _id: FrLo doi: 10.52202/085713-1444 external_id: arxiv: - '2505.17708' fulldoi: https://doi.org/10.52202/085713-1444 has_accepted_license: '1' intvolume: ' 38' language: - iso: eng main_file_link: - open_access: '1' url: https://doi.org/10.48550/arXiv.2505.17708 month: '12' oa: 1 oa_version: Preprint project: - _id: a392d8f0-b034-11f1-b88e-d3025c6734f6 grant_number: COE12 name: Bilateral Artificial Intelligence (Locatello) publication: 39th Annual Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' related_material: link: - relation: software url: https://github.com/shimenghuang/a-measurement-perspective-of-crl status: public title: The third pillar of causal analysis? A measurement perspective on causal representations tmp: image: /images/cc_by.png legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0) short: CC BY (4.0) type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 38 year: '2025' ... --- OA_place: repository OA_type: green _id: '21070' abstract: - lang: eng text: 'Deep learning systems deployed in real-world applications often encounter data that is different from their in-distribution (ID). A reliable model should ideally abstain from making decisions in this out-of-distribution (OOD) setting. Existing state-of-the-art methods primarily focus on feature distances, such as k-th nearest neighbors and distances to decision boundaries, either overlooking or ineffectively using in-distribution statistics. In this work, we propose a novel angle-based metric for OOD detection that is computed relative to the in-distribution structure. We demonstrate that the angles between feature representations and decision boundaries, viewed from the mean of in-distribution features, serve as an effective discriminative factor between ID and OOD data. We evaluate our method on nine ImageNet-pretrained models. Our approach achieves the lowest FPR in 5 out of 9 ImageNet models, obtains the best average FPR overall, and consistently ranking among the top 3 across all evaluated models. Furthermore, we highlight the benefits of contrastive representations by showing strong performance with ResNet SCL and CLIP architectures. Finally, we demonstrate that the scale-invariant nature of our score enables an ensemble strategy via simple score summation. ' acknowledgement: "This research was funded in whole or in part by the Austrian Science Fund (FWF) 10.55776/COE12. For open access purposes, the author has applied a CC BY public copyright license to any accepted manuscript version arising from this submission.\r\n" alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Berker full_name: Demirel, Berker id: 8b4bc47f-3200-11ee-973b-8f0e7be21a9f last_name: Demirel - first_name: 'Marco ' full_name: 'Fumero, Marco ' last_name: Fumero - first_name: Francesco full_name: Locatello, Francesco id: 26cfd52f-2483-11ee-8040-88983bcc06d4 last_name: Locatello orcid: 0000-0002-4850-0683 citation: ama: 'Demirel B, Fumero M, Locatello F. Out-of-Distribution detection with relative angles. In: 39th Annual Conference on Neural Information Processing Systems. Vol 38. Neural Information Processing Systems Foundation; 2025. doi:10.52202/085713-3125' apa: 'Demirel, B., Fumero, M., & Locatello, F. (2025). Out-of-Distribution detection with relative angles. In 39th Annual Conference on Neural Information Processing Systems (Vol. 38). San Diego, CA, United States: Neural Information Processing Systems Foundation. https://doi.org/10.52202/085713-3125' chicago: Demirel, Berker, Marco Fumero, and Francesco Locatello. “Out-of-Distribution Detection with Relative Angles.” In 39th Annual Conference on Neural Information Processing Systems, Vol. 38. Neural Information Processing Systems Foundation, 2025. https://doi.org/10.52202/085713-3125. ieee: B. Demirel, M. Fumero, and F. Locatello, “Out-of-Distribution detection with relative angles,” in 39th Annual Conference on Neural Information Processing Systems, San Diego, CA, United States, 2025, vol. 38. ista: 'Demirel B, Fumero M, Locatello F. 2025. Out-of-Distribution detection with relative angles. 39th Annual Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 38.' mla: Demirel, Berker, et al. “Out-of-Distribution Detection with Relative Angles.” 39th Annual Conference on Neural Information Processing Systems, vol. 38, Neural Information Processing Systems Foundation, 2025, doi:10.52202/085713-3125. short: B. Demirel, M. Fumero, F. Locatello, in:, 39th Annual Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2025. conference: end_date: 2025-12-07 location: San Diego, CA, United States name: 'NeurIPS: Neural Information Processing Systems' start_date: 2025-12-02 corr_author: '1' das_tickbox: '1' date_created: 2026-01-29T14:26:47Z date_published: 2025-12-01T00:00:00Z date_updated: 2026-10-15T09:32:34Z day: '01' ddc: - '000' department: - _id: FrLo doi: 10.52202/085713-3125 external_id: arxiv: - '2410.04525' fulldoi: https://doi.org/10.52202/085713-3125 has_accepted_license: '1' intvolume: ' 38' language: - iso: eng main_file_link: - open_access: '1' url: https://doi.org/10.48550/arXiv.2410.04525 month: '12' oa: 1 oa_version: Preprint project: - _id: a392d8f0-b034-11f1-b88e-d3025c6734f6 grant_number: COE12 name: Bilateral Artificial Intelligence (Locatello) publication: 39th Annual Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' related_material: link: - relation: software url: https://github.com/berkerdemirel/ORA-OOD-Detection-with-Relative-Angles status: public title: Out-of-Distribution detection with relative angles tmp: image: /images/cc_by.png legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0) short: CC BY (4.0) type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 38 year: '2025' ... --- OA_place: publisher OA_type: free access _id: '22823' abstract: - lang: eng text: This position paper argues that setting the privacy budget in differential privacy should not be viewed as an important limitation of differential privacy compared to alternative methods for privacy-preserving machine learning. The so-called problem of interpreting the privacy budget is often presented as a major hindrance to the wider adoption of differential privacy in real-world deployments and is sometimes used to promote alternative mitigation techniques for data protection. We believe this misleads decision-makers into choosing unsafe methods. We argue that the difficulty in interpreting privacy budgets does not stem from the definition of differential privacy itself, but from the intrinsic difficulty of estimating privacy risks in context, a challenge that any rigorous method for privacy risk assessment face. Moreover, we claim that any sound method for estimating privacy risks should, given the current state of research, be expressible within the differential privacy framework or justify why it cannot. acknowledgement: "This research was funded in whole by the Austrian Science Fund (FWF) 10.55776/COE12. I thank\r\nAlberto Naibo for the discussion and his invitation to \"Logique, droit, IA : penser les algorithmes\"\r\ncolloquium. I thank Simone Bombari for the discussions. I thank EurIPS for the active poster session." alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No author: - first_name: Edwige Audrey Lucienne full_name: Cyffers, Edwige Audrey Lucienne id: 20d4c299-977a-11ef-ae55-98b15ac64a57 last_name: Cyffers citation: ama: 'Cyffers EAL. Setting ε is not the issue in differential privacy. In: 39th Conference on Neural Information Processing Systems. Vol 38. Neural Information Processing Systems Foundation; 2025:159016-159028. doi:10.52202/085713-4798' apa: 'Cyffers, E. A. L. (2025). Setting ε is not the issue in differential privacy. In 39th Conference on Neural Information Processing Systems (Vol. 38, pp. 159016–159028). San Diego, CA, United States: Neural Information Processing Systems Foundation. https://doi.org/10.52202/085713-4798' chicago: Cyffers, Edwige Audrey Lucienne. “Setting ε Is Not the Issue in Differential Privacy.” In 39th Conference on Neural Information Processing Systems, 38:159016–28. Neural Information Processing Systems Foundation, 2025. https://doi.org/10.52202/085713-4798. ieee: E. A. L. Cyffers, “Setting ε is not the issue in differential privacy,” in 39th Conference on Neural Information Processing Systems, San Diego, CA, United States, 2025, vol. 38, pp. 159016–159028. ista: 'Cyffers EAL. 2025. Setting ε is not the issue in differential privacy. 39th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 38, 159016–159028.' mla: Cyffers, Edwige Audrey Lucienne. “Setting ε Is Not the Issue in Differential Privacy.” 39th Conference on Neural Information Processing Systems, vol. 38, Neural Information Processing Systems Foundation, 2025, pp. 159016–28, doi:10.52202/085713-4798. short: E.A.L. Cyffers, in:, 39th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2025, pp. 159016–159028. conference: end_date: 2025-12-07 location: San Diego, CA, United States name: 'NeurIPS: Neural Information Processing Systems' start_date: 2025-12-02 corr_author: '1' das_tickbox: '0' date_created: 2026-09-06T22:01:59Z date_published: 2025-12-02T00:00:00Z date_updated: 2026-10-15T09:34:50Z day: '02' department: - _id: ChLa doi: 10.52202/085713-4798 fulldoi: https://doi.org/10.52202/085713-4798 intvolume: ' 38' language: - iso: eng main_file_link: - open_access: '1' url: https://doi.org/10.52202/085713-4798 month: '12' oa: 1 oa_version: Published Version page: 159016-159028 project: - _id: d8f03aaa-b035-11f1-8588-d5147fa879e0 grant_number: COE12 name: Bilateral Artificial Intelligence (Lampert) publication: 39th Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 isbn: - '9798331338275' publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' researchdata_availability: no scopus_import: '1' status: public supplementarymaterial: yes title: Setting ε is not the issue in differential privacy type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 38 year: '2025' ... --- OA_place: publisher OA_type: free access _id: '18890' abstract: - lang: eng text: Deep Neural Collapse (DNC) refers to the surprisingly rigid structure of the data representations in the final layers of Deep Neural Networks (DNNs). Though the phenomenon has been measured in a variety of settings, its emergence is typically explained via data-agnostic approaches, such as the unconstrained features model. In this work, we introduce a data-dependent setting where DNC forms due to feature learning through the average gradient outer product (AGOP). The AGOP is defined with respect to a learned predictor and is equal to the uncentered covariance matrix of its input-output gradients averaged over the training dataset. The Deep Recursive Feature Machine (Deep RFM) is a method that constructs a neural network by iteratively mapping the data with the AGOP and applying an untrained random feature map. We demonstrate empirically that DNC occurs in Deep RFM across standard settings as a consequence of the projection with the AGOP matrix computed at each layer. Further, we theoretically explain DNC in Deep RFM in an asymptotic setting and as a result of kernel learning. We then provide evidence that this mechanism holds for neural networks more generally. In particular, we show that the right singular vectors and values of the weights can be responsible for the majority of within-class variability collapse for DNNs trained in the feature learning regime. As observed in recent work, this singular structure is highly correlated with that of the AGOP. acknowledgement: 'We acknowledge support from the National Science Foundation (NSF) and the Simons Foundation for the Collaboration on the Theoretical Foundations of Deep Learning through awards DMS-2031883 and #814639 as well as the TILOS institute (NSF CCF-2112665). This work used the programs (1) XSEDE (Extreme science and engineering discovery environment) which is supported by NSF grant numbers ACI-1548562, and (2) ACCESS (Advanced cyberinfrastructure coordination ecosystem: services & support) which is supported by NSF grants numbers #2138259, #2138286, #2138307, #2137603, and #2138296. Specifically, we used the resources from SDSC Expanse GPU compute nodes, and NCSA Delta system, via allocations TG-CIS220009. Marco Mondelli is supported by the 2019 Lopez-Loreta prize. We also acknowledge useful feedback from anonymous reviewers. ' alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Daniel full_name: Beaglehole, Daniel last_name: Beaglehole - first_name: Peter full_name: Súkeník, Peter id: d64d6a8d-eb8e-11eb-b029-96fd216dec3c last_name: Súkeník - first_name: Marco full_name: Mondelli, Marco id: 27EB676C-8706-11E9-9510-7717E6697425 last_name: Mondelli orcid: 0000-0002-3242-7020 - first_name: Mikhail full_name: Belkin, Mikhail last_name: Belkin citation: ama: 'Beaglehole D, Súkeník P, Mondelli M, Belkin M. Average gradient outer product as a mechanism for deep neural collapse. In: 38th Annual Conference on Neural Information Processing Systems. Vol 37. Neural Information Processing Systems Foundation; 2024. doi:10.52202/079017-4156' apa: 'Beaglehole, D., Súkeník, P., Mondelli, M., & Belkin, M. (2024). Average gradient outer product as a mechanism for deep neural collapse. In 38th Annual Conference on Neural Information Processing Systems (Vol. 37). Vancouver, Canada: Neural Information Processing Systems Foundation. https://doi.org/10.52202/079017-4156' chicago: Beaglehole, Daniel, Peter Súkeník, Marco Mondelli, and Mikhail Belkin. “Average Gradient Outer Product as a Mechanism for Deep Neural Collapse.” In 38th Annual Conference on Neural Information Processing Systems, Vol. 37. Neural Information Processing Systems Foundation, 2024. https://doi.org/10.52202/079017-4156. ieee: D. Beaglehole, P. Súkeník, M. Mondelli, and M. Belkin, “Average gradient outer product as a mechanism for deep neural collapse,” in 38th Annual Conference on Neural Information Processing Systems, Vancouver, Canada, 2024, vol. 37. ista: 'Beaglehole D, Súkeník P, Mondelli M, Belkin M. 2024. Average gradient outer product as a mechanism for deep neural collapse. 38th Annual Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 37.' mla: Beaglehole, Daniel, et al. “Average Gradient Outer Product as a Mechanism for Deep Neural Collapse.” 38th Annual Conference on Neural Information Processing Systems, vol. 37, Neural Information Processing Systems Foundation, 2024, doi:10.52202/079017-4156. short: D. Beaglehole, P. Súkeník, M. Mondelli, M. Belkin, in:, 38th Annual Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2024. conference: end_date: 2024-12-16 location: Vancouver, Canada name: 'NeurIPS: Neural Information Processing Systems' start_date: 2024-12-16 corr_author: '1' das_tickbox: '0' date_created: 2025-01-27T11:11:40Z date_published: 2024-12-01T00:00:00Z date_updated: 2026-10-15T09:39:46Z day: '01' department: - _id: GradSch - _id: MaMo doi: 10.52202/079017-4156 external_id: arxiv: - '2402.13728' fulldoi: https://doi.org/10.52202/079017-4156 intvolume: ' 37' language: - iso: eng main_file_link: - open_access: '1' url: https://doi.org/10.52202/079017-4156 month: '12' oa: 1 oa_version: Published Version project: - _id: 059876FA-7A3F-11EA-A408-12923DDC885E name: Prix Lopez-Loretta 2019 - Marco Mondelli publication: 38th Annual Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' researchdata_availability: no scopus_import: '1' status: public supplementarymaterial: no title: Average gradient outer product as a mechanism for deep neural collapse type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 37 year: '2024' ... --- OA_place: publisher OA_type: free access _id: '18875' abstract: - lang: eng text: Current state-of-the-art methods for differentially private model training are based on matrix factorization techniques. However, these methods suffer from high computational overhead because they require numerically solving a demanding optimization problem to determine an approximately optimal factorization prior to the actual model training. In this work, we present a new matrix factorization approach, BSR, which overcomes this computational bottleneck. By exploiting properties of the standard matrix square root, BSR allows to efficiently handle also large-scale problems. For the key scenario of stochastic gradient descent with momentum and weight decay, we even derive analytical expressions for BSR that render the computational overhead negligible. We prove bounds on the approximation quality that hold both in the centralized and in the federated learning setting. Our numerical experiments demonstrate that models trained using BSR perform on par with the best existing methods, while completely avoiding their computational overhead. alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Nikita full_name: Kalinin, Nikita id: 4b14526e-14d2-11ed-ba64-c14c9553d137 last_name: Kalinin - first_name: Christoph full_name: Lampert, Christoph id: 40C20FD2-F248-11E8-B48F-1D18A9856A87 last_name: Lampert orcid: 0000-0001-8622-7887 citation: ama: 'Kalinin N, Lampert C. Banded square root matrix factorization for differentially private model training. In: 38th Annual Conference on Neural Information Processing Systems. Vol 37. Neural Information Processing Systems Foundation; 2024. doi:10.52202/079017-0560' apa: 'Kalinin, N., & Lampert, C. (2024). Banded square root matrix factorization for differentially private model training. In 38th Annual Conference on Neural Information Processing Systems (Vol. 37). Vancouver, Canada: Neural Information Processing Systems Foundation. https://doi.org/10.52202/079017-0560' chicago: Kalinin, Nikita, and Christoph Lampert. “Banded Square Root Matrix Factorization for Differentially Private Model Training.” In 38th Annual Conference on Neural Information Processing Systems, Vol. 37. Neural Information Processing Systems Foundation, 2024. https://doi.org/10.52202/079017-0560. ieee: N. Kalinin and C. Lampert, “Banded square root matrix factorization for differentially private model training,” in 38th Annual Conference on Neural Information Processing Systems, Vancouver, Canada, 2024, vol. 37. ista: 'Kalinin N, Lampert C. 2024. Banded square root matrix factorization for differentially private model training. 38th Annual Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 37.' mla: Kalinin, Nikita, and Christoph Lampert. “Banded Square Root Matrix Factorization for Differentially Private Model Training.” 38th Annual Conference on Neural Information Processing Systems, vol. 37, Neural Information Processing Systems Foundation, 2024, doi:10.52202/079017-0560. short: N. Kalinin, C. Lampert, in:, 38th Annual Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2024. conference: end_date: 2024-12-16 location: Vancouver, Canada name: 'NeurIPS: Neural Information Processing Systems' start_date: 2024-12-16 corr_author: '1' das_tickbox: '0' date_created: 2025-01-24T17:58:16Z date_published: 2024-12-01T00:00:00Z date_updated: 2026-10-15T09:38:13Z day: '01' ddc: - '000' department: - _id: GradSch - _id: ChLa doi: 10.52202/079017-0560 external_id: arxiv: - '2405.13763' fulldoi: https://doi.org/10.52202/079017-0560 has_accepted_license: '1' intvolume: ' 37' language: - iso: eng main_file_link: - open_access: '1' url: https://doi.org/10.52202/079017-0560 month: '12' oa: 1 oa_version: Published Version publication: 38th Annual Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' researchdata_availability: no scopus_import: '1' status: public supplementarymaterial: no title: Banded square root matrix factorization for differentially private model training type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 37 year: '2024' ... --- OA_place: repository OA_type: green _id: '18996' abstract: - lang: eng text: 'We consider the linear causal representation learning setting where we observe a linear mixing of d unknown latent factors, which follow a linear structural causal model. Recent work has shown that it is possible to recover the latent factors as well as the underlying structural causal model over them, up to permutation and scaling, provided that we have at least d environments, each of which corresponds to perfect interventions on a single latent node (factor). After this powerful result, a key open problem faced by the community has been to relax these conditions: allow for coarser than perfect single-node interventions, and allow for fewer than d of them, since the number of latent factors d could be very large. In this work, we consider precisely such a setting, where we allow a smaller than d number of environments, and also allow for very coarse interventions that can very coarsely \textit{change the entire causal graph over the latent factors}. On the flip side, we relax what we wish to extract to simply the \textit{list of nodes that have shifted between one or more environments}. We provide a surprising identifiability result that it is indeed possible, under some very mild standard assumptions, to identify the set of shifted nodes. Our identifiability proof moreover is a constructive one: we explicitly provide necessary and sufficient conditions for a node to be a shifted node, and show that we can check these conditions given observed data. Our algorithm lends itself very naturally to the sample setting where instead of just interventional distributions, we are provided datasets of samples from each of these distributions. We corroborate our results on both synthetic experiments as well as an interesting psychometric dataset. The code can be found at https://github.com/TianyuCodings/iLCS.' alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Tianyu full_name: Chen, Tianyu last_name: Chen - first_name: Kevin full_name: Bello, Kevin last_name: Bello - first_name: Francesco full_name: Locatello, Francesco id: 26cfd52f-2483-11ee-8040-88983bcc06d4 last_name: Locatello orcid: 0000-0002-4850-0683 - first_name: Bryon full_name: Aragam, Bryon last_name: Aragam - first_name: Pradeep Kumar full_name: Ravikumar, Pradeep Kumar last_name: Ravikumar citation: ama: 'Chen T, Bello K, Locatello F, Aragam B, Ravikumar PK. Identifying general mechanism shifts in linear causal representations. In: 38th Conference on Neural Information Processing Systems. Vol 37. Neural Information Processing Systems Foundation; 2024. doi:10.52202/079017-1343' apa: 'Chen, T., Bello, K., Locatello, F., Aragam, B., & Ravikumar, P. K. (2024). Identifying general mechanism shifts in linear causal representations. In 38th Conference on Neural Information Processing Systems (Vol. 37). Vancouver, Canada: Neural Information Processing Systems Foundation. https://doi.org/10.52202/079017-1343' chicago: Chen, Tianyu, Kevin Bello, Francesco Locatello, Bryon Aragam, and Pradeep Kumar Ravikumar. “Identifying General Mechanism Shifts in Linear Causal Representations.” In 38th Conference on Neural Information Processing Systems, Vol. 37. Neural Information Processing Systems Foundation, 2024. https://doi.org/10.52202/079017-1343. ieee: T. Chen, K. Bello, F. Locatello, B. Aragam, and P. K. Ravikumar, “Identifying general mechanism shifts in linear causal representations,” in 38th Conference on Neural Information Processing Systems, Vancouver, Canada, 2024, vol. 37. ista: 'Chen T, Bello K, Locatello F, Aragam B, Ravikumar PK. 2024. Identifying general mechanism shifts in linear causal representations. 38th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 37.' mla: Chen, Tianyu, et al. “Identifying General Mechanism Shifts in Linear Causal Representations.” 38th Conference on Neural Information Processing Systems, vol. 37, Neural Information Processing Systems Foundation, 2024, doi:10.52202/079017-1343. short: T. Chen, K. Bello, F. Locatello, B. Aragam, P.K. Ravikumar, in:, 38th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2024. conference: end_date: 2024-12-16 location: Vancouver, Canada name: 'NeurIPS: Neural Information Processing Systems' start_date: 2024-12-16 das_tickbox: '0' date_created: 2025-02-04T13:09:34Z date_published: 2024-09-25T00:00:00Z date_updated: 2026-10-15T09:44:18Z day: '25' ddc: - '000' department: - _id: FrLo doi: 10.52202/079017-1343 external_id: arxiv: - '2410.24059' file: - access_level: open_access checksum: 75c3091e70bd2916cd94afbf40a0c425 content_type: application/pdf creator: dernst date_created: 2025-02-04T13:09:08Z date_updated: 2025-02-04T13:09:08Z file_id: '18997' file_name: 2024_NeurIPS_Chen.pdf file_size: 5659119 relation: main_file success: 1 file_date_updated: 2025-02-04T13:09:08Z fulldoi: https://doi.org/10.52202/079017-1343 has_accepted_license: '1' intvolume: ' 37' language: - iso: eng month: '09' oa: 1 oa_version: Published Version publication: 38th Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' researchdata_availability: no scopus_import: '1' status: public supplementarymaterial: no title: Identifying general mechanism shifts in linear causal representations tmp: image: /images/cc_by.png legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0) short: CC BY (4.0) type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 37 year: '2024' ... --- OA_place: publisher OA_type: free access _id: '19005' abstract: - lang: eng text: "Causal representation learning promises to extend causal models to hidden causal\r\nvariables from raw entangled measurements. However, most progress has focused\r\non proving identifiability results in different settings, and we are not aware of any\r\nsuccessful real-world application. At the same time, the field of dynamical systems\r\nbenefited from deep learning and scaled to countless applications but does not allow\r\nparameter identification. In this paper, we draw a clear connection between the two\r\nand their key assumptions, allowing us to apply identifiable methods developed\r\nin causal representation learning to dynamical systems. At the same time, we can\r\nleverage scalable differentiable solvers developed for differential equations to build\r\nmodels that are both identifiable and practical. Overall, we learn explicitly controllable models that isolate the trajectory-specific parameters for further downstream\r\ntasks such as out-of-distribution classification or treatment effect estimation. We\r\nexperiment with a wind simulator with partially known factors of variation. We\r\nalso apply the resulting model to real-world climate data and successfully answer\r\ndownstream causal questions in line with existing literature on climate change.\r\nCode is available at https://github.com/CausalLearningAI/crl-dynamical-systems." acknowledgement: "We thank Niklas Boers for recommending the SpeedyWeather simulator and Valentino Maiorca\r\nfor guidance on Fourier transformation for SST data. We are also grateful to Shimeng Huang and Riccardo Cadei for their feedback on the treatment effect estimation experiment and to Jiale Chen and Adeel Pervez for their assistance with the solver implementation. Finally, we appreciate the anonymous reviewers for their insightful suggestions, which helped improve the manuscript. " alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Dingling full_name: Yao, Dingling id: d3e02e50-48a8-11ee-8f62-c108061797fa last_name: Yao - first_name: Caroline J full_name: Muller, Caroline J id: f978ccb0-3f7f-11eb-b193-b0e2bd13182b last_name: Muller orcid: 0000-0001-5836-5350 - first_name: Francesco full_name: Locatello, Francesco id: 26cfd52f-2483-11ee-8040-88983bcc06d4 last_name: Locatello orcid: 0000-0002-4850-0683 citation: ama: 'Yao D, Muller CJ, Locatello F. Marrying causal representation learning with dynamical systems for science. In: 38th Conference on Neural Information Processing Systems. Vol 37. Neural Information Processing Systems Foundation; 2024. doi:10.52202/079017-2290' apa: 'Yao, D., Muller, C. J., & Locatello, F. (2024). Marrying causal representation learning with dynamical systems for science. In 38th Conference on Neural Information Processing Systems (Vol. 37). Vancouver, Canada: Neural Information Processing Systems Foundation. https://doi.org/10.52202/079017-2290' chicago: Yao, Dingling, Caroline J Muller, and Francesco Locatello. “Marrying Causal Representation Learning with Dynamical Systems for Science.” In 38th Conference on Neural Information Processing Systems, Vol. 37. Neural Information Processing Systems Foundation, 2024. https://doi.org/10.52202/079017-2290. ieee: D. Yao, C. J. Muller, and F. Locatello, “Marrying causal representation learning with dynamical systems for science,” in 38th Conference on Neural Information Processing Systems, Vancouver, Canada, 2024, vol. 37. ista: 'Yao D, Muller CJ, Locatello F. 2024. Marrying causal representation learning with dynamical systems for science. 38th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 37.' mla: Yao, Dingling, et al. “Marrying Causal Representation Learning with Dynamical Systems for Science.” 38th Conference on Neural Information Processing Systems, vol. 37, Neural Information Processing Systems Foundation, 2024, doi:10.52202/079017-2290. short: D. Yao, C.J. Muller, F. Locatello, in:, 38th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2024. conference: end_date: 2024-12-16 location: Vancouver, Canada name: 'NeurIPS: Neural Information Processing Systems' start_date: 2024-12-16 corr_author: '1' das_tickbox: '0' date_created: 2025-02-05T07:49:00Z date_published: 2024-12-01T00:00:00Z date_updated: 2026-10-15T09:47:17Z day: '01' ddc: - '000' - '550' department: - _id: CaMu - _id: FrLo doi: 10.52202/079017-2290 external_id: arxiv: - '2405.13888' file: - access_level: open_access checksum: fe8832367e7143876f178244385d859e content_type: application/pdf creator: dernst date_created: 2025-02-05T07:44:58Z date_updated: 2025-02-05T07:44:58Z file_id: '19006' file_name: 2024_NeurIPS_Yao.pdf file_size: 2595855 relation: main_file success: 1 file_date_updated: 2025-02-05T07:44:58Z fulldoi: https://doi.org/10.52202/079017-2290 has_accepted_license: '1' intvolume: ' 37' language: - iso: eng month: '12' oa: 1 oa_version: Published Version publication: 38th Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' related_material: link: - relation: software url: https://github.com/CausalLearningAI/crl-dynamical-systems researchdata_availability: no scopus_import: '1' status: public supplementarymaterial: no title: Marrying causal representation learning with dynamical systems for science type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 37 year: '2024' ... --- OA_place: repository OA_type: green _id: '19512' abstract: - lang: eng text: "Differential privacy with gradual expiration models the setting where data items\r\narrive in a stream and at a given time t the privacy loss guaranteed for a data item\r\nseen at time (t − d) is εg(d), where g is a monotonically non-decreasing function.\r\nWe study the fundamental continual (binary) counting problem where each data\r\nitem consists of a bit, and the algorithm needs to output at each time step the sum of\r\nall the bits streamed so far. For a stream of length T and privacy without expiration\r\ncontinual counting is possible with maximum (over all time steps) additive error\r\nO(log2\r\n(T)/ε) and the best known lower bound is Ω(log(T)/ε); closing this gap\r\nis a challenging open problem.\r\nWe show that the situation is very different for privacy with gradual expiration by\r\ngiving upper and lower bounds for a large set of expiration functions g. Specifically,\r\nour algorithm achieves an additive error of O(log(T)/ε) for a large set of privacy\r\nexpiration functions. We also give a lower bound that shows that if C is the additive\r\nerror of any ε-DP algorithm for this problem, then the product of C and the privacy\r\nexpiration function after 2C steps must be Ω(log(T)/ε). Our algorithm matches\r\nthis lower bound as its additive error is O(log(T)/ε), even when g(2C) = O(1).\r\nOur empirical evaluation shows that we achieve a slowly growing privacy loss\r\nwith significantly smaller empirical privacy loss for large values of d than a natural\r\nbaseline algorithm." acknowledgement: 'Monika Henzinger: This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No. 101019564) and the Austrian Science Fund (FWF) grant DOI 10.55776/Z422, grant DOI 10.55776/I5982, and grant DOI 10.55776/P33775 with additional funding from the netidee SCIENCE Stiftung, 2020–2024. Joel Daniel Andersson and Rasmus Pagh are affiliated with Basic Algorithms Research Copenhagen (BARC), supported by the VILLUM Foundation grant 16582, and are also supported by Providentia, a Data Science Distinguished Investigator grant from Novo Nordisk Fonden. Teresa Anna Steiner is supported by a research grant (VIL51463) from VILLUM FONDEN. This work was done while Teresa Anna Steiner was a Postdoc at the Technical University of Denmark. Jalaj Upadhyay’s research was funded by the Rutgers Decanal Grant no. 302918 and an unrestricted gift from Google.' alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Joel Daniel full_name: Andersson, Joel Daniel last_name: Andersson - first_name: Monika H full_name: Henzinger, Monika H id: 540c9bbd-f2de-11ec-812d-d04a5be85630 last_name: Henzinger orcid: 0000-0002-5008-6530 - first_name: Rasmus full_name: Pagh, Rasmus last_name: Pagh - first_name: Teresa Anna full_name: Steiner, Teresa Anna last_name: Steiner - first_name: Jalaj full_name: Upadhyay, Jalaj last_name: Upadhyay citation: ama: 'Andersson JD, Henzinger M, Pagh R, Steiner TA, Upadhyay J. Continual counting with gradual privacy expiration. In: 38th Conference on Neural Information Processing Systems. Vol 37. Neural Information Processing Systems Foundation; 2024. doi:10.52202/079017-0299' apa: 'Andersson, J. D., Henzinger, M., Pagh, R., Steiner, T. A., & Upadhyay, J. (2024). Continual counting with gradual privacy expiration. In 38th Conference on Neural Information Processing Systems (Vol. 37). Vancouver, Canada: Neural Information Processing Systems Foundation. https://doi.org/10.52202/079017-0299' chicago: Andersson, Joel Daniel, Monika Henzinger, Rasmus Pagh, Teresa Anna Steiner, and Jalaj Upadhyay. “Continual Counting with Gradual Privacy Expiration.” In 38th Conference on Neural Information Processing Systems, Vol. 37. Neural Information Processing Systems Foundation, 2024. https://doi.org/10.52202/079017-0299. ieee: J. D. Andersson, M. Henzinger, R. Pagh, T. A. Steiner, and J. Upadhyay, “Continual counting with gradual privacy expiration,” in 38th Conference on Neural Information Processing Systems, Vancouver, Canada, 2024, vol. 37. ista: 'Andersson JD, Henzinger M, Pagh R, Steiner TA, Upadhyay J. 2024. Continual counting with gradual privacy expiration. 38th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 37.' mla: Andersson, Joel Daniel, et al. “Continual Counting with Gradual Privacy Expiration.” 38th Conference on Neural Information Processing Systems, vol. 37, Neural Information Processing Systems Foundation, 2024, doi:10.52202/079017-0299. short: J.D. Andersson, M. Henzinger, R. Pagh, T.A. Steiner, J. Upadhyay, in:, 38th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2024. conference: end_date: 2024-12-15 location: Vancouver, Canada name: 'NeurIPS: Neural Information Processing Systems' start_date: 2024-12-09 corr_author: '1' date_created: 2025-04-06T22:01:32Z date_published: 2024-12-20T00:00:00Z date_updated: 2026-10-15T09:52:07Z day: '20' department: - _id: MoHe doi: 10.52202/079017-0299 ec_funded: 1 external_id: arxiv: - '2406.03802' fulldoi: https://doi.org/10.52202/079017-0299 intvolume: ' 37' language: - iso: eng main_file_link: - open_access: '1' url: https://doi.org/10.48550/arXiv.2406.03802 month: '12' oa: 1 oa_version: Preprint project: - _id: bd9ca328-d553-11ed-ba76-dc4f890cfe62 call_identifier: H2020 grant_number: '101019564' name: The design and evaluation of modern fully dynamic data structures - _id: 34def286-11ca-11ed-8bc3-da5948e1613c grant_number: Z00422 name: Efficient algorithms - _id: bda196b2-d553-11ed-ba76-8e8ee6c21103 grant_number: I05982 name: Static and Dynamic Hierarchical Graph Decompositions - _id: bd9e3a2e-d553-11ed-ba76-8aa684ce17fe grant_number: P33775 name: Fast Algorithms for a Reactive Network Layer publication: 38th Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' scopus_import: '1' status: public title: Continual counting with gradual privacy expiration type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 37 year: '2024' ... --- OA_place: repository OA_type: green _id: '19515' abstract: - lang: eng text: "Neural models learn data representations that lie on low-dimensional manifolds,\r\nyet modeling the relation between these representational spaces is an ongoing challenge. By integrating spectral geometry principles into neural modeling, we show\r\nthat this problem can be better addressed in the functional domain, mitigating complexity, while enhancing interpretability and performances on downstream tasks.\r\nTo this end, we introduce a multi-purpose framework to the representation learning\r\ncommunity, which allows to: (i) compare different spaces in an interpretable way\r\nand measure their intrinsic similarity; (ii) find correspondences between them, both\r\nin unsupervised and weakly supervised settings, and (iii) to effectively transfer\r\nrepresentations between distinct spaces. We validate our framework on various\r\napplications, ranging from stitching to retrieval tasks, and on multiple modalities,\r\ndemonstrating that Latent Functional Maps can serve as a swiss-army knife for\r\nrepresentation alignment" acknowledgement: MF is supported by the MSCA IST-Bridge fellowship which has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 101034413. ER and VM are supported by the PNRR MUR project PE0000013-FAIR. MP is supported by the Sapienza grant "Predicting and Explaining Clinical Trial Outcomes", prot. RG12218166FA3F13. alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Marco full_name: Fumero, Marco id: 1c1593eb-393f-11ef-bb8e-ab4f1e979650 last_name: Fumero - first_name: Marco full_name: Pegoraro, Marco last_name: Pegoraro - first_name: Valentino full_name: Maiorca, Valentino last_name: Maiorca - first_name: Francesco full_name: Locatello, Francesco id: 26cfd52f-2483-11ee-8040-88983bcc06d4 last_name: Locatello orcid: 0000-0002-4850-0683 - first_name: Emanuele full_name: Rodolà, Emanuele last_name: Rodolà citation: ama: 'Fumero M, Pegoraro M, Maiorca V, Locatello F, Rodolà E. Latent functional maps: A spectral framework for representation alignment. In: 38th Conference on Neural Information Processing Systems. Vol 37. Neural Information Processing Systems Foundation; 2024. doi:10.52202/079017-2115' apa: 'Fumero, M., Pegoraro, M., Maiorca, V., Locatello, F., & Rodolà, E. (2024). Latent functional maps: A spectral framework for representation alignment. In 38th Conference on Neural Information Processing Systems (Vol. 37). Vancouver, Canada: Neural Information Processing Systems Foundation. https://doi.org/10.52202/079017-2115' chicago: 'Fumero, Marco, Marco Pegoraro, Valentino Maiorca, Francesco Locatello, and Emanuele Rodolà. “Latent Functional Maps: A Spectral Framework for Representation Alignment.” In 38th Conference on Neural Information Processing Systems, Vol. 37. Neural Information Processing Systems Foundation, 2024. https://doi.org/10.52202/079017-2115.' ieee: 'M. Fumero, M. Pegoraro, V. Maiorca, F. Locatello, and E. Rodolà, “Latent functional maps: A spectral framework for representation alignment,” in 38th Conference on Neural Information Processing Systems, Vancouver, Canada, 2024, vol. 37.' ista: 'Fumero M, Pegoraro M, Maiorca V, Locatello F, Rodolà E. 2024. Latent functional maps: A spectral framework for representation alignment. 38th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 37.' mla: 'Fumero, Marco, et al. “Latent Functional Maps: A Spectral Framework for Representation Alignment.” 38th Conference on Neural Information Processing Systems, vol. 37, Neural Information Processing Systems Foundation, 2024, doi:10.52202/079017-2115.' short: M. Fumero, M. Pegoraro, V. Maiorca, F. Locatello, E. Rodolà, in:, 38th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2024. conference: end_date: 2024-12-15 location: Vancouver, Canada name: 'NeurIPS: Neural Information Processing Systems' start_date: 2024-12-09 corr_author: '1' date_created: 2025-04-06T22:01:32Z date_published: 2024-12-20T00:00:00Z date_updated: 2026-10-15T09:52:55Z day: '20' department: - _id: FrLo doi: 10.52202/079017-2115 ec_funded: 1 external_id: arxiv: - '2406.14183' fulldoi: https://doi.org/10.52202/079017-2115 intvolume: ' 37' language: - iso: eng main_file_link: - open_access: '1' url: https://doi.org/10.48550/arXiv.2406.14183 month: '12' oa: 1 oa_version: Preprint project: - _id: fc2ed2f7-9c52-11eb-aca3-c01059dda49c call_identifier: H2020 grant_number: '101034413' name: 'IST-BRIDGE: International postdoctoral program' publication: 38th Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' scopus_import: '1' status: public title: 'Latent functional maps: A spectral framework for representation alignment' type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 37 year: '2024' ... --- OA_place: repository OA_type: green _id: '19509' abstract: - lang: eng text: "Machine Learning and AI have the potential to transform data-driven scientific discovery,\r\nenabling accurate predictions for several scientific phenomena. As many scientific questions\r\nare inherently causal, this paper looks at the causal inference task of treatment effect\r\nestimation, where the outcome of interest is recorded in high-dimensional observations\r\nin a Randomized Controlled Trial (RCT). Despite being the simplest possible causal setting\r\nand a perfect fit for deep learning, we theoretically find that many common choices in the\r\nliterature may lead to biased estimates. To test the practical impact of these considerations,\r\nwe recorded ISTAnt, the first real-world benchmark for causal inference downstream tasks\r\non high-dimensional observations as an RCT studying how garden ants (Lasius neglectus)\r\nrespond to microparticles applied onto their colony members by hygienic grooming.\r\nComparing 6 480 models fine-tuned from state-of-the-art visual backbones, we find that\r\nthe sampling and modeling choices significantly affect the accuracy of the causal estimate,\r\nand that classification accuracy is not a proxy thereof. We further validated the analysis,\r\nrepeating it on a synthetically generated visual data set controlling the causal model. Our\r\nresults suggest that future benchmarks should carefully consider real downstream scientific\r\nquestions, especially causal ones. Further, we highlight guidelines for representation\r\nlearning methods to help answer causal questions in the sciences.\r\nCode: https://github.com/CausalLearningAI/ISTAnt\r\nData: https://doi.org/10.6084/m9.figshare.26484934.v2" acknowledgement: We thank Piersilvio De Bartolomeis, and the full Causal Learning and Artificial Intelligence (CLAI) group at ISTA for the extremely helpful discussions. Riccardo Cadei was supported by a Google Research Scholar Award and a Google Initiated Gift to Francesco Locatello. We thank the Social Immunity team at ISTA, particularly Michaela Hönigsberger and Wilfrid Jean Louis, for supporting the ecological experiment and Farnaz Beikzadeh Abbasi, Luisa Fiebig and Martin Estermann for annotating ant behavior in ISTAnt. alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Riccardo full_name: Cadei, Riccardo id: 0fa8b76f-72f0-11ef-b75a-a5da96e5ad6b last_name: Cadei - first_name: Lukas full_name: Lindorfer, Lukas id: 85f0e6d3-06b3-11ec-8982-8c5049fa4455 last_name: Lindorfer - first_name: Sylvia full_name: Cremer, Sylvia id: 2F64EC8C-F248-11E8-B48F-1D18A9856A87 last_name: Cremer orcid: 0000-0002-2193-3868 - first_name: Cordelia full_name: Schmid, Cordelia last_name: Schmid - first_name: Francesco full_name: Locatello, Francesco id: 26cfd52f-2483-11ee-8040-88983bcc06d4 last_name: Locatello orcid: 0000-0002-4850-0683 citation: ama: 'Cadei R, Lindorfer L, Cremer S, Schmid C, Locatello F. Smoke and mirrors in causal downstream tasks. In: 38th Conference on Neural Information Processing Systems. Vol 37. Neural Information Processing Systems Foundation; 2024. doi:10.52202/079017-0820' apa: 'Cadei, R., Lindorfer, L., Cremer, S., Schmid, C., & Locatello, F. (2024). Smoke and mirrors in causal downstream tasks. In 38th Conference on Neural Information Processing Systems (Vol. 37). Vancouver, Canada: Neural Information Processing Systems Foundation. https://doi.org/10.52202/079017-0820' chicago: Cadei, Riccardo, Lukas Lindorfer, Sylvia Cremer, Cordelia Schmid, and Francesco Locatello. “Smoke and Mirrors in Causal Downstream Tasks.” In 38th Conference on Neural Information Processing Systems, Vol. 37. Neural Information Processing Systems Foundation, 2024. https://doi.org/10.52202/079017-0820. ieee: R. Cadei, L. Lindorfer, S. Cremer, C. Schmid, and F. Locatello, “Smoke and mirrors in causal downstream tasks,” in 38th Conference on Neural Information Processing Systems, Vancouver, Canada, 2024, vol. 37. ista: 'Cadei R, Lindorfer L, Cremer S, Schmid C, Locatello F. 2024. Smoke and mirrors in causal downstream tasks. 38th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 37.' mla: Cadei, Riccardo, et al. “Smoke and Mirrors in Causal Downstream Tasks.” 38th Conference on Neural Information Processing Systems, vol. 37, Neural Information Processing Systems Foundation, 2024, doi:10.52202/079017-0820. short: R. Cadei, L. Lindorfer, S. Cremer, C. Schmid, F. Locatello, in:, 38th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2024. conference: end_date: 2024-12-15 location: Vancouver, Canada name: 'NeurIPS: Neural Information Processing Systems' start_date: 2024-12-09 das_tickbox: '1' dataavailabilitystatement: "Code: https://github.com/CausalLearningAI/ISTAnt\r\nData: https://doi.org/10.6084/m9.figshare.26484934.v2" date_created: 2025-04-06T22:01:32Z date_published: 2024-12-20T00:00:00Z date_updated: 2026-10-21T10:52:44Z day: '20' department: - _id: SyCr - _id: FrLo doi: 10.52202/079017-0820 external_id: arxiv: - '2405.17151' fulldoi: https://doi.org/10.52202/079017-0820 intvolume: ' 37' language: - iso: eng main_file_link: - open_access: '1' url: https://doi.org/10.48550/arXiv.2405.17151 month: '12' oa: 1 oa_version: Preprint publication: 38th Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' related_material: link: - relation: software url: https://github.com/CausalLearningAI/ISTAnt record: - id: '18847' relation: earlier_version status: public researchdata_availability: yes scopus_import: '1' status: public supplementarymaterial: yes title: Smoke and mirrors in causal downstream tasks type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 37 year: '2024' ... --- _id: '14921' abstract: - lang: eng text: Neural collapse (NC) refers to the surprising structure of the last layer of deep neural networks in the terminal phase of gradient descent training. Recently, an increasing amount of experimental evidence has pointed to the propagation of NC to earlier layers of neural networks. However, while the NC in the last layer is well studied theoretically, much less is known about its multi-layered counterpart - deep neural collapse (DNC). In particular, existing work focuses either on linear layers or only on the last two layers at the price of an extra assumption. Our paper fills this gap by generalizing the established analytical framework for NC - the unconstrained features model - to multiple non-linear layers. Our key technical contribution is to show that, in a deep unconstrained features model, the unique global optimum for binary classification exhibits all the properties typical of DNC. This explains the existing experimental evidence of DNC. We also empirically show that (i) by optimizing deep unconstrained features models via gradient descent, the resulting solution agrees well with our theory, and (ii) trained networks recover the unconstrained features suitable for the occurrence of DNC, thus supporting the validity of this modeling principle. acknowledgement: M. M. is partially supported by the 2019 Lopez-Loreta Prize. The authors would like to thank Eugenia Iofinova, Bernd Prach and Simone Bombari for valuable feedback on the manuscript. alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Peter full_name: Súkeník, Peter id: d64d6a8d-eb8e-11eb-b029-96fd216dec3c last_name: Súkeník - first_name: Marco full_name: Mondelli, Marco id: 27EB676C-8706-11E9-9510-7717E6697425 last_name: Mondelli orcid: 0000-0002-3242-7020 - first_name: Christoph full_name: Lampert, Christoph id: 40C20FD2-F248-11E8-B48F-1D18A9856A87 last_name: Lampert orcid: 0000-0001-8622-7887 citation: ama: 'Súkeník P, Mondelli M, Lampert C. Deep neural collapse is provably optimal for the deep unconstrained features model. In: 37th Conference on Neural Information Processing Systems. Neural Information Processing Systems Foundation; 2023.' apa: 'Súkeník, P., Mondelli, M., & Lampert, C. (2023). Deep neural collapse is provably optimal for the deep unconstrained features model. In 37th Conference on Neural Information Processing Systems. New Orleans, LA, United States: Neural Information Processing Systems Foundation.' chicago: Súkeník, Peter, Marco Mondelli, and Christoph Lampert. “Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model.” In 37th Conference on Neural Information Processing Systems. Neural Information Processing Systems Foundation, 2023. ieee: P. Súkeník, M. Mondelli, and C. Lampert, “Deep neural collapse is provably optimal for the deep unconstrained features model,” in 37th Conference on Neural Information Processing Systems, New Orleans, LA, United States, 2023. ista: 'Súkeník P, Mondelli M, Lampert C. 2023. Deep neural collapse is provably optimal for the deep unconstrained features model. 37th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, .' mla: Súkeník, Peter, et al. “Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model.” 37th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2023. short: P. Súkeník, M. Mondelli, C. Lampert, in:, 37th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2023. conference: end_date: 2023-12-16 location: New Orleans, LA, United States name: 'NeurIPS: Neural Information Processing Systems' start_date: 2023-12-10 corr_author: '1' das_tickbox: '1' date_created: 2024-02-02T11:17:41Z date_published: 2023-12-15T00:00:00Z date_updated: 2026-07-07T06:35:57Z day: '15' department: - _id: MaMo - _id: ChLa external_id: arxiv: - '2305.13165' language: - iso: eng main_file_link: - open_access: '1' url: ' https://doi.org/10.48550/arXiv.2305.13165' month: '12' oa: 1 oa_version: Preprint project: - _id: 059876FA-7A3F-11EA-A408-12923DDC885E name: Prix Lopez-Loretta 2019 - Marco Mondelli publication: 37th Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' status: public title: Deep neural collapse is provably optimal for the deep unconstrained features model type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 year: '2023' ... --- OA_place: publisher OA_type: gold _id: '14954' abstract: - lang: eng text: "When domain knowledge is limited and experimentation is restricted by ethical,\r\nfinancial, or time constraints, practitioners turn to observational causal discovery\r\nmethods to recover the causal structure, exploiting the statistical properties of their\r\ndata. Because causal discovery without further assumptions is an ill-posed problem,\r\neach algorithm comes with its own set of usually untestable assumptions, some\r\nof which are hard to meet in real datasets. Motivated by these considerations, this\r\npaper extensively benchmarks the empirical performance of recent causal discovery\r\nmethods on observational iid data generated under different background conditions,\r\nallowing for violations of the critical assumptions required by each selected approach. Our experimental findings show that score matching-based methods demonstrate surprising performance in the false positive and false negative rate of the\r\ninferred graph in these challenging scenarios, and we provide theoretical insights\r\ninto their performance. This work is also the first effort to benchmark the stability of\r\ncausal discovery algorithms with respect to the values of their hyperparameters. Finally, we hope this paper will set a new standard for the evaluation of causal discovery methods and can serve as an accessible entry point for practitioners interested\r\nin the field, highlighting the empirical implications of different algorithm choices." acknowledgement: "We thank Kun Zhang and Carl-Johann Simon-Gabriel for the insightful discussions. This work\r\nhas been supported by AFOSR, grant n. FA8655-20-1-7035. FM is supported by Programma\r\nOperativo Nazionale ricerca e innovazione 2014-2020. FM partially contributed to this work during an internship at Amazon Web Services with FL. FL partially contributed while at AWS." alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Francesco full_name: Montagna, Francesco last_name: Montagna - first_name: Atalanti A. full_name: Mastakouri, Atalanti A. last_name: Mastakouri - first_name: Elias full_name: Eulig, Elias last_name: Eulig - first_name: Nicoletta full_name: Noceti, Nicoletta last_name: Noceti - first_name: Lorenzo full_name: Rosasco, Lorenzo last_name: Rosasco - first_name: Dominik full_name: Janzing, Dominik last_name: Janzing - first_name: Bryon full_name: Aragam, Bryon last_name: Aragam - first_name: Francesco full_name: Locatello, Francesco id: 26cfd52f-2483-11ee-8040-88983bcc06d4 last_name: Locatello orcid: 0000-0002-4850-0683 citation: ama: 'Montagna F, Mastakouri AA, Eulig E, et al. Assumption violations in causal discovery and the robustness of score matching. In: 37th Conference on Neural Information Processing Systems. Vol 36. Neural Information Processing Systems Foundation; 2023. doi:10.52202/075280-2050' apa: 'Montagna, F., Mastakouri, A. A., Eulig, E., Noceti, N., Rosasco, L., Janzing, D., … Locatello, F. (2023). Assumption violations in causal discovery and the robustness of score matching. In 37th Conference on Neural Information Processing Systems (Vol. 36). New Orleans, LA, United States: Neural Information Processing Systems Foundation. https://doi.org/10.52202/075280-2050' chicago: Montagna, Francesco, Atalanti A. Mastakouri, Elias Eulig, Nicoletta Noceti, Lorenzo Rosasco, Dominik Janzing, Bryon Aragam, and Francesco Locatello. “Assumption Violations in Causal Discovery and the Robustness of Score Matching.” In 37th Conference on Neural Information Processing Systems, Vol. 36. Neural Information Processing Systems Foundation, 2023. https://doi.org/10.52202/075280-2050. ieee: F. Montagna et al., “Assumption violations in causal discovery and the robustness of score matching,” in 37th Conference on Neural Information Processing Systems, New Orleans, LA, United States, 2023, vol. 36. ista: 'Montagna F, Mastakouri AA, Eulig E, Noceti N, Rosasco L, Janzing D, Aragam B, Locatello F. 2023. Assumption violations in causal discovery and the robustness of score matching. 37th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 36.' mla: Montagna, Francesco, et al. “Assumption Violations in Causal Discovery and the Robustness of Score Matching.” 37th Conference on Neural Information Processing Systems, vol. 36, Neural Information Processing Systems Foundation, 2023, doi:10.52202/075280-2050. short: F. Montagna, A.A. Mastakouri, E. Eulig, N. Noceti, L. Rosasco, D. Janzing, B. Aragam, F. Locatello, in:, 37th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2023. conference: end_date: 2023-12-16 location: New Orleans, LA, United States name: 'NeurIPS: Neural Information Processing Systems' start_date: 2023-12-12 das_tickbox: '0' date_created: 2024-02-07T15:11:56Z date_published: 2023-12-20T00:00:00Z date_updated: 2026-09-17T12:59:55Z day: '20' ddc: - '000' department: - _id: FrLo doi: 10.52202/075280-2050 external_id: arxiv: - '2310.13387' file: - access_level: open_access checksum: 71d38402b4edef4c084b39f3e2f04526 content_type: application/pdf creator: dernst date_created: 2026-08-13T07:49:38Z date_updated: 2026-08-13T07:49:38Z file_id: '22701' file_name: 2023_Neurips_Montagna.pdf file_size: 7640984 relation: main_file success: 1 file_date_updated: 2026-08-13T07:49:38Z fulldoi: https://doi.org/10.52202/075280-2050 has_accepted_license: '1' intvolume: ' 36' language: - iso: eng month: '12' oa: 1 oa_version: Published Version publication: 37th Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' researchdata_availability: no status: public supplementarymaterial: yes title: Assumption violations in causal discovery and the robustness of score matching type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 36 year: '2023' ... --- _id: '15023' abstract: - lang: eng text: Reinforcement learning has shown promising results in learning neural network policies for complicated control tasks. However, the lack of formal guarantees about the behavior of such policies remains an impediment to their deployment. We propose a novel method for learning a composition of neural network policies in stochastic environments, along with a formal certificate which guarantees that a specification over the policy's behavior is satisfied with the desired probability. Unlike prior work on verifiable RL, our approach leverages the compositional nature of logical specifications provided in SpectRL, to learn over graphs of probabilistic reach-avoid specifications. The formal guarantees are provided by learning neural network policies together with reach-avoid supermartingales (RASM) for the graph’s sub-tasks and then composing them into a global policy. We also derive a tighter lower bound compared to previous work on the probability of reach-avoidance implied by a RASM, which is required to find a compositional policy with an acceptable probabilistic threshold for complex tasks with multiple edge policies. We implement a prototype of our approach and evaluate it on a Stochastic Nine Rooms environment. acknowledgement: "This work was supported in part by the ERC-2020-AdG 101020093 (VAMOS) and the ERC-2020-\r\nCoG 863818 (FoRM-SMArt)." alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Dorde full_name: Zikelic, Dorde id: 294AA7A6-F248-11E8-B48F-1D18A9856A87 last_name: Zikelic orcid: 0000-0002-4681-1699 - first_name: Mathias full_name: Lechner, Mathias id: 3DC22916-F248-11E8-B48F-1D18A9856A87 last_name: Lechner - first_name: Abhinav full_name: Verma, Abhinav id: a235593c-d7fa-11eb-a0c5-b22ca3c66ee6 last_name: Verma - first_name: Krishnendu full_name: Chatterjee, Krishnendu id: 2E5DCA20-F248-11E8-B48F-1D18A9856A87 last_name: Chatterjee orcid: 0000-0002-4561-241X - first_name: Thomas A full_name: Henzinger, Thomas A id: 40876CD8-F248-11E8-B48F-1D18A9856A87 last_name: Henzinger orcid: 0000-0002-2985-7724 citation: ama: 'Zikelic D, Lechner M, Verma A, Chatterjee K, Henzinger TA. Compositional policy learning in stochastic control systems with formal guarantees. In: 37th Conference on Neural Information Processing Systems. Neural Information Processing Systems Foundation; 2023.' apa: 'Zikelic, D., Lechner, M., Verma, A., Chatterjee, K., & Henzinger, T. A. (2023). Compositional policy learning in stochastic control systems with formal guarantees. In 37th Conference on Neural Information Processing Systems. New Orleans, LA, United States: Neural Information Processing Systems Foundation.' chicago: Zikelic, Dorde, Mathias Lechner, Abhinav Verma, Krishnendu Chatterjee, and Thomas A Henzinger. “Compositional Policy Learning in Stochastic Control Systems with Formal Guarantees.” In 37th Conference on Neural Information Processing Systems. Neural Information Processing Systems Foundation, 2023. ieee: D. Zikelic, M. Lechner, A. Verma, K. Chatterjee, and T. A. Henzinger, “Compositional policy learning in stochastic control systems with formal guarantees,” in 37th Conference on Neural Information Processing Systems, New Orleans, LA, United States, 2023. ista: 'Zikelic D, Lechner M, Verma A, Chatterjee K, Henzinger TA. 2023. Compositional policy learning in stochastic control systems with formal guarantees. 37th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, .' mla: Zikelic, Dorde, et al. “Compositional Policy Learning in Stochastic Control Systems with Formal Guarantees.” 37th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2023. short: D. Zikelic, M. Lechner, A. Verma, K. Chatterjee, T.A. Henzinger, in:, 37th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2023. conference: end_date: 2023-12-16 location: New Orleans, LA, United States name: 'NeurIPS: Neural Information Processing Systems' start_date: 2023-12-10 corr_author: '1' das_tickbox: '1' date_created: 2024-02-25T09:23:24Z date_published: 2023-12-15T00:00:00Z date_updated: 2026-09-17T12:59:12Z day: '15' ddc: - '000' department: - _id: ToHe - _id: KrCh ec_funded: 1 external_id: arxiv: - '2312.01456' file: - access_level: open_access checksum: 739c6d72506b778302d4e708723bf12c content_type: application/pdf creator: dernst date_created: 2024-07-22T11:45:17Z date_updated: 2024-07-22T11:45:17Z file_id: '17309' file_name: 2023_NeurIPS_Zikelic.pdf file_size: 562008 relation: main_file success: 1 file_date_updated: 2024-07-22T11:45:17Z has_accepted_license: '1' language: - iso: eng month: '12' oa: 1 oa_version: Published Version project: - _id: 0599E47C-7A3F-11EA-A408-12923DDC885E call_identifier: H2020 grant_number: '863818' name: 'Formal Methods for Stochastic Models: Algorithms and Applications' - _id: 62781420-2b32-11ec-9570-8d9b63373d4d call_identifier: H2020 grant_number: '101020093' name: Vigilant Algorithmic Monitoring of Software publication: 37th Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' status: public title: Compositional policy learning in stochastic control systems with formal guarantees type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 year: '2023' ... --- OA_place: publisher OA_type: gold _id: '14953' abstract: - lang: eng text: "This paper provides statistical sample complexity bounds for score-matching and\r\nits applications in causal discovery. We demonstrate that accurate estimation of the\r\nscore function is achievable by training a standard deep ReLU neural network using\r\nstochastic gradient descent. We establish bounds on the error rate of recovering\r\ncausal relationships using the score-matching-based causal discovery method of\r\nRolland et al. [2022], assuming a sufficiently good estimation of the score function.\r\nFinally, we analyze the upper bound of score-matching estimation within the scorebased generative modeling, which has been applied for causal discovery but is also\r\nof independent interest within the domain of generative models.y" acknowledgement: "We are thankful to the reviewers for providing constructive feedback and Kun Zhang and Dominik\r\nJanzing for helpful discussion on the special case of deterministic children. This work was supported\r\nby Hasler Foundation Program: Hasler Responsible AI (project number 21043). This work was\r\nsupported by the Swiss National Science Foundation (SNSF) under grant number 200021_205011.\r\nFrancesco Locatello did not contribute to this work at Amazon. " alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Zhenyu full_name: Zhu, Zhenyu last_name: Zhu - first_name: Francesco full_name: Locatello, Francesco id: 26cfd52f-2483-11ee-8040-88983bcc06d4 last_name: Locatello orcid: 0000-0002-4850-0683 - first_name: Volkan full_name: Cevher, Volkan last_name: Cevher citation: ama: 'Zhu Z, Locatello F, Cevher V. Sample complexity bounds for score-matching: Causal discovery and generative modeling. In: 37th Conference on Neural Information Processing Systems. Vol 36. Neural Information Processing Systems Foundation; 2023:3325-3337. doi:10.52202/075280-0147' apa: 'Zhu, Z., Locatello, F., & Cevher, V. (2023). Sample complexity bounds for score-matching: Causal discovery and generative modeling. In 37th Conference on Neural Information Processing Systems (Vol. 36, pp. 3325–3337). New Orleans, LA, United States: Neural Information Processing Systems Foundation. https://doi.org/10.52202/075280-0147' chicago: 'Zhu, Zhenyu, Francesco Locatello, and Volkan Cevher. “Sample Complexity Bounds for Score-Matching: Causal Discovery and Generative Modeling.” In 37th Conference on Neural Information Processing Systems, 36:3325–37. Neural Information Processing Systems Foundation, 2023. https://doi.org/10.52202/075280-0147.' ieee: 'Z. Zhu, F. Locatello, and V. Cevher, “Sample complexity bounds for score-matching: Causal discovery and generative modeling,” in 37th Conference on Neural Information Processing Systems, New Orleans, LA, United States, 2023, vol. 36, pp. 3325–3337.' ista: 'Zhu Z, Locatello F, Cevher V. 2023. Sample complexity bounds for score-matching: Causal discovery and generative modeling. 37th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 36, 3325–3337.' mla: 'Zhu, Zhenyu, et al. “Sample Complexity Bounds for Score-Matching: Causal Discovery and Generative Modeling.” 37th Conference on Neural Information Processing Systems, vol. 36, Neural Information Processing Systems Foundation, 2023, pp. 3325–37, doi:10.52202/075280-0147.' short: Z. Zhu, F. Locatello, V. Cevher, in:, 37th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2023, pp. 3325–3337. conference: end_date: 2023-12-16 location: New Orleans, LA, United States name: 'NeurIPS: Neural Information Processing Systems' start_date: 2023-12-12 corr_author: '1' das_tickbox: '0' date_created: 2024-02-07T15:11:11Z date_published: 2023-10-27T00:00:00Z date_updated: 2026-09-17T12:59:35Z day: '27' ddc: - '000' department: - _id: FrLo doi: 10.52202/075280-0147 external_id: arxiv: - '2310.18123' file: - access_level: open_access checksum: 6f10adadefceb3bb50ad0b2637493381 content_type: application/pdf creator: dernst date_created: 2026-08-13T07:02:51Z date_updated: 2026-08-13T07:02:51Z file_id: '22700' file_name: 2023_Neurips_Zhu.pdf file_size: 305362 relation: main_file success: 1 file_date_updated: 2026-08-13T07:02:51Z fulldoi: https://doi.org/10.52202/075280-0147 has_accepted_license: '1' intvolume: ' 36' language: - iso: eng month: '10' oa: 1 oa_version: Published Version page: 3325-3337 publication: 37th Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 isbn: - '9781713899921' publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' researchdata_availability: no scopus_import: '1' status: public supplementarymaterial: yes title: 'Sample complexity bounds for score-matching: Causal discovery and generative modeling' type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 36 year: '2023' ... --- OA_place: repository OA_type: green _id: '12537' abstract: - lang: eng text: 'The Neural Tangent Kernel (NTK) has emerged as a powerful tool to provide memorization, optimization and generalization guarantees in deep neural networks. A line of work has studied the NTK spectrum for two-layer and deep networks with at least a layer with Ω(N) neurons, N being the number of training samples. Furthermore, there is increasing evidence suggesting that deep networks with sub-linear layer widths are powerful memorizers and optimizers, as long as the number of parameters exceeds the number of samples. Thus, a natural open question is whether the NTK is well conditioned in such a challenging sub-linear setup. In this paper, we answer this question in the affirmative. Our key technical contribution is a lower bound on the smallest NTK eigenvalue for deep networks with the minimum possible over-parameterization: the number of parameters is roughly Ω(N) and, hence, the number of neurons is as little as Ω(N−−√). To showcase the applicability of our NTK bounds, we provide two results concerning memorization capacity and optimization guarantees for gradient descent training.' acknowledgement: "The authors were partially supported by the 2019 Lopez-Loreta prize, and they would like to thank\r\nQuynh Nguyen, Mahdi Soltanolkotabi and Adel Javanmard for helpful discussions.\r\n" alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Simone full_name: Bombari, Simone id: ca726dda-de17-11ea-bc14-f9da834f63aa last_name: Bombari - first_name: Mohammad Hossein full_name: Amani, Mohammad Hossein last_name: Amani - first_name: Marco full_name: Mondelli, Marco id: 27EB676C-8706-11E9-9510-7717E6697425 last_name: Mondelli orcid: 0000-0002-3242-7020 citation: ama: 'Bombari S, Amani MH, Mondelli M. Memorization and optimization in deep neural networks with minimum over-parameterization. In: 36th Conference on Neural Information Processing Systems. Vol 35. Neural Information Processing Systems Foundation; 2022:7628-7640.' apa: 'Bombari, S., Amani, M. H., & Mondelli, M. (2022). Memorization and optimization in deep neural networks with minimum over-parameterization. In 36th Conference on Neural Information Processing Systems (Vol. 35, pp. 7628–7640). New Orleans, LA, United States: Neural Information Processing Systems Foundation.' chicago: Bombari, Simone, Mohammad Hossein Amani, and Marco Mondelli. “Memorization and Optimization in Deep Neural Networks with Minimum Over-Parameterization.” In 36th Conference on Neural Information Processing Systems, 35:7628–40. Neural Information Processing Systems Foundation, 2022. ieee: S. Bombari, M. H. Amani, and M. Mondelli, “Memorization and optimization in deep neural networks with minimum over-parameterization,” in 36th Conference on Neural Information Processing Systems, New Orleans, LA, United States, 2022, vol. 35, pp. 7628–7640. ista: 'Bombari S, Amani MH, Mondelli M. 2022. Memorization and optimization in deep neural networks with minimum over-parameterization. 36th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 35, 7628–7640.' mla: Bombari, Simone, et al. “Memorization and Optimization in Deep Neural Networks with Minimum Over-Parameterization.” 36th Conference on Neural Information Processing Systems, vol. 35, Neural Information Processing Systems Foundation, 2022, pp. 7628–40. short: S. Bombari, M.H. Amani, M. Mondelli, in:, 36th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2022, pp. 7628–7640. conference: end_date: 2022-12-09 location: New Orleans, LA, United States name: 'NeurIPS: Neural Information Processing Systems' start_date: 2022-11-28 corr_author: '1' date_created: 2023-02-10T13:46:37Z date_published: 2022-07-24T00:00:00Z date_updated: 2026-09-21T13:07:01Z day: '24' department: - _id: MaMo external_id: arxiv: - '2205.10217' intvolume: ' 35' language: - iso: eng main_file_link: - open_access: '1' url: ' https://doi.org/10.48550/arXiv.2205.10217' month: '07' oa: 1 oa_version: Preprint page: 7628-7640 project: - _id: 059876FA-7A3F-11EA-A408-12923DDC885E name: Prix Lopez-Loretta 2019 - Marco Mondelli publication: 36th Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 isbn: - '9781713871088' publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' related_material: record: - id: '22857' relation: dissertation_contains status: public status: public title: Memorization and optimization in deep neural networks with minimum over-parameterization type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 35 year: '2022' ... --- _id: '14326' abstract: - lang: eng text: "Learning object-centric representations of complex scenes is a promising step towards enabling efficient abstract reasoning from low-level perceptual features. Yet, most deep learning approaches learn distributed representations that do not capture the compositional properties of natural scenes. In this paper, we present the Slot Attention module, an architectural component that interfaces with perceptual representations such as the output of a convolutional neural network and produces a set of task-dependent abstract representations which we call slots. These slots are exchangeable and can bind to any object in the input by specializing through a competitive procedure over multiple rounds of attention. We empirically demonstrate that Slot Attention can extract object-centric representations that enable generalization to unseen compositions when trained on unsupervised object discovery and supervised property prediction tasks.\r\n\r\n" alternative_title: - Advances in Neural Information Processing Systems article_processing_charge: No arxiv: 1 author: - first_name: Francesco full_name: Locatello, Francesco id: 26cfd52f-2483-11ee-8040-88983bcc06d4 last_name: Locatello orcid: 0000-0002-4850-0683 - first_name: Dirk full_name: Weissenborn, Dirk last_name: Weissenborn - first_name: Thomas full_name: Unterthiner, Thomas last_name: Unterthiner - first_name: Aravindh full_name: Mahendran, Aravindh last_name: Mahendran - first_name: Georg full_name: Heigold, Georg last_name: Heigold - first_name: Jakob full_name: Uszkoreit, Jakob last_name: Uszkoreit - first_name: Alexey full_name: Dosovitskiy, Alexey last_name: Dosovitskiy - first_name: Thomas full_name: Kipf, Thomas last_name: Kipf citation: ama: 'Locatello F, Weissenborn D, Unterthiner T, et al. Object-centric learning with slot attention. In: 34th International Conference on Neural Information Processing Systems. Vol 33. Neural Information Processing Systems Foundation; 2020:11525-11538.' apa: 'Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., … Kipf, T. (2020). Object-centric learning with slot attention. In 34th International Conference on Neural Information Processing Systems (Vol. 33, pp. 11525–11538). Virtual: Neural Information Processing Systems Foundation.' chicago: Locatello, Francesco, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. “Object-Centric Learning with Slot Attention.” In 34th International Conference on Neural Information Processing Systems, 33:11525–38. Neural Information Processing Systems Foundation, 2020. ieee: F. Locatello et al., “Object-centric learning with slot attention,” in 34th International Conference on Neural Information Processing Systems, Virtual, 2020, vol. 33, pp. 11525–11538. ista: 'Locatello F, Weissenborn D, Unterthiner T, Mahendran A, Heigold G, Uszkoreit J, Dosovitskiy A, Kipf T. 2020. Object-centric learning with slot attention. 34th International Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 33, 11525–11538.' mla: Locatello, Francesco, et al. “Object-Centric Learning with Slot Attention.” 34th International Conference on Neural Information Processing Systems, vol. 33, Neural Information Processing Systems Foundation, 2020, pp. 11525–38. short: F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, T. Kipf, in:, 34th International Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2020, pp. 11525–11538. conference: end_date: 2020-12-12 location: Virtual name: 'NeurIPS: Neural Information Processing Systems' start_date: 2020-12-06 date_created: 2023-09-13T12:03:46Z date_published: 2020-12-20T00:00:00Z date_updated: 2025-07-10T11:50:47Z day: '20' department: - _id: FrLo extern: '1' external_id: arxiv: - '2006.15055' intvolume: ' 33' language: - iso: eng main_file_link: - open_access: '1' url: https://doi.org/10.48550/arXiv.2006.15055 month: '12' oa: 1 oa_version: Preprint page: 11525-11538 publication: 34th International Conference on Neural Information Processing Systems publication_identifier: eissn: - 1049-5258 isbn: - '9781713829546' publication_status: published publisher: Neural Information Processing Systems Foundation quality_controlled: '1' status: public title: Object-centric learning with slot attention type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 33 year: '2020' ...