{"oa":1,"month":"12","title":"Head pursuit: Probing attention specialization in multimodal transformers","language":[{"iso":"eng"}],"abstract":[{"text":"Language and vision-language models have shown impressive performance across a wide range of tasks, but their internal mechanisms remain only partly understood. In this work, we study how individual attention heads in text-generative models specialize in specific semantic or visual attributes. Building on an established interpretability method, we reinterpret the practice of probing intermediate activations with the final decoding layer through the lens of signal processing. This lets us analyze multiple samples in a principled way and rank attention heads based on their relevance to target concepts. Our results show consistent patterns of specialization at the head level across both unimodal and multimodal transformers. Remarkably, we find that editing as few as 1% of the heads, selected using our method, can reliably suppress or enhance targeted concepts in the model output. We validate our approach on language tasks such as question answering and toxicity mitigation, as well as vision-language tasks including image classification and captioning. Our findings highlight an interpretable and controllable structure within attention layers, offering simple tools for understanding and editing large-scale generative models.","lang":"eng"}],"OA_type":"gold","OA_place":"publisher","acknowledgement":"The authors acknowledge the Area Science Park supercomputing platform ORFEO made available for conducting the research reported in this paper, and the technical support of the Laboratory of Data Engineering staff. LB, DD and AC were supported by the project “Supporto alla diagnosi di malattie rare tramite l’intelligenza artificiale\" CUP: F53C22001770002 and “Valutazione automatica delle immagini diagnostiche tramite l’intelligenza artificiale\", CUP: F53C22001780002. LB was supported by the European Union – NextGenerationEU within the project PNRR “Finanziamento di progetti presentati da giovani ricercatori\" - Mission 4 Component 2 Investment 1.2, CUP: J93C25000440001. AC was supported by the European Union – NextGenerationEU within the project PNRR “PRP@CERIC\" IR0000028 - Mission 4 Component 2 Investment 3.1 Action 3.1.1. ","file_date_updated":"2026-01-29T14:29:14Z","status":"public","date_published":"2025-12-15T00:00:00Z","quality_controlled":"1","main_file_link":[{"url":"https://doi.org/10.48550/arXiv.2510.21518","open_access":"1"}],"tmp":{"name":"Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)","image":"/images/cc_by.png","short":"CC BY (4.0)","legal_code_url":"https://creativecommons.org/licenses/by/4.0/legalcode"},"citation":{"ama":"Basile L, Maiorca V, Doimo D, Locatello F, Cazzaniga A. Head pursuit: Probing attention specialization in multimodal transformers. In: <i>39th Annual Conference on Neural Information Processing Systems</i>. Vol 38. Neural Information Processing Systems Foundation; 2025.","short":"L. Basile, V. Maiorca, D. Doimo, F. Locatello, A. Cazzaniga, in:, 39th Annual Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2025.","chicago":"Basile, Lorenzo, Valentino Maiorca, Diego Doimo, Francesco Locatello, and Alberto Cazzaniga. “Head Pursuit: Probing Attention Specialization in Multimodal Transformers.” In <i>39th Annual Conference on Neural Information Processing Systems</i>, Vol. 38. Neural Information Processing Systems Foundation, 2025.","mla":"Basile, Lorenzo, et al. “Head Pursuit: Probing Attention Specialization in Multimodal Transformers.” <i>39th Annual Conference on Neural Information Processing Systems</i>, vol. 38, Neural Information Processing Systems Foundation, 2025.","apa":"Basile, L., Maiorca, V., Doimo, D., Locatello, F., &#38; Cazzaniga, A. (2025). Head pursuit: Probing attention specialization in multimodal transformers. In <i>39th Annual Conference on Neural Information Processing Systems</i> (Vol. 38). San Diego, CA, United States: Neural Information Processing Systems Foundation.","ieee":"L. Basile, V. Maiorca, D. Doimo, F. Locatello, and A. Cazzaniga, “Head pursuit: Probing attention specialization in multimodal transformers,” in <i>39th Annual Conference on Neural Information Processing Systems</i>, San Diego, CA, United States, 2025, vol. 38.","ista":"Basile L, Maiorca V, Doimo D, Locatello F, Cazzaniga A. 2025. Head pursuit: Probing attention specialization in multimodal transformers. 39th Annual Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems vol. 38."},"file":[{"date_updated":"2026-01-29T14:29:14Z","file_id":"21073","date_created":"2026-01-29T14:29:14Z","content_type":"application/pdf","access_level":"open_access","creator":"flocatel","success":1,"file_name":"2510.21518v2.pdf","checksum":"85be3f98663e2595cf37001852b477cb","file_size":4271547,"relation":"main_file"}],"_id":"21072","arxiv":1,"ddc":["000"],"oa_version":"Preprint","publisher":"Neural Information Processing Systems Foundation","year":"2025","has_accepted_license":"1","external_id":{"arxiv":["2510.21518"]},"publication_identifier":{"issn":["1049-5258"]},"publication_status":"epub_ahead","department":[{"_id":"FrLo"}],"volume":38,"date_updated":"2026-02-11T08:55:36Z","date_created":"2026-01-29T14:29:23Z","conference":{"location":"San Diego, CA, United States","start_date":"2025-12-02","end_date":"2025-12-07","name":"NeurIPS: Neural Information Processing Systems"},"article_processing_charge":"No","publication":"39th Annual Conference on Neural Information Processing Systems","type":"conference","author":[{"full_name":"Basile, Lorenzo","first_name":"Lorenzo","last_name":"Basile"},{"last_name":"Maiorca","first_name":"Valentino","full_name":"Maiorca, Valentino"},{"first_name":"Diego","last_name":"Doimo","full_name":"Doimo, Diego"},{"orcid":"0000-0002-4850-0683","id":"26cfd52f-2483-11ee-8040-88983bcc06d4","last_name":"Locatello","first_name":"Francesco","full_name":"Locatello, Francesco"},{"first_name":"Alberto","last_name":"Cazzaniga","full_name":"Cazzaniga, Alberto"}],"day":"15","intvolume":"        38","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87"}