ASIF: Coupled data turns unimodal models to multimodal without training

conference paper ASIF: Coupled data turns unimodal models to multimodal without training Advances in Neural Information Processing Systems published yes Antonio Norelli author Marco Fumero author Valentino Maiorca author Luca Moschella author Emanuele Rodolà author Francesco Locatello author 26cfd52f-2483-11ee-8040-88983bcc06d40000-0002-4850-0683 FrLo department NeurIPS: Neural Information Processing Systems CLIP proved that aligning visual and language spaces is key to solving many vision tasks without explicit training, but required to train image and text encoders from scratch on a huge dataset. LiT improved this by only training the text encoder and using a pre-trained vision network. In this paper, we show that a common space can be created without any training at all, using single-domain encoders (trained with or without supervision) and a much smaller amount of image-text pairs. Furthermore, our model has unique properties. Most notably, deploying a new version with updated training samples can be done in a matter of seconds. Additionally, the representations in the common space are easily interpretable as every dimension corresponds to the similarity of the input to a unique entry in the multimodal dataset. Experiments on standard zero-shot visual benchmarks demonstrate the typical transfer ability of image-text models. Overall, our method represents a simple yet surprisingly strong baseline for foundation multi-modal models, raising important questions on their data efficiency and on the role of retrieval in machine learning. https://research-explorer.ista.ac.at/download/14216/18994/2023_NeurIPS_Fumero.pdf application/pdfno Neural Information Processing Systems Foundation2023New Orleans, LA, United States eng 37th Conference on Neural Information Processing Systems 9781713899921 2210.01738 3615303-15319 https://github.com/noranta4/ASIF Norelli A, Fumero M, Maiorca V, Moschella L, Rodolà E, Locatello F. ASIF: Coupled data turns unimodal models to multimodal without training. In: 37th Conference on Neural Information Processing Systems. Vol 36. Neural Information Processing Systems Foundation; 2023:15303-15319. A. Norelli, M. Fumero, V. Maiorca, L. Moschella, E. Rodolà, and F. Locatello, “ASIF: Coupled data turns unimodal models to multimodal without training,” in 37th Conference on Neural Information Processing Systems, New Orleans, LA, United States, 2023, vol. 36, pp. 15303–15319. Norelli, Antonio, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà, and Francesco Locatello. “ASIF: Coupled Data Turns Unimodal Models to Multimodal without Training.” In 37th Conference on Neural Information Processing Systems, 36:15303–19. Neural Information Processing Systems Foundation, 2023. A. Norelli, M. Fumero, V. Maiorca, L. Moschella, E. Rodolà, F. Locatello, in:, 37th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2023, pp. 15303–15319. Norelli A, Fumero M, Maiorca V, Moschella L, Rodolà E, Locatello F. 2023. ASIF: Coupled data turns unimodal models to multimodal without training. 37th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 36, 15303–15319. Norelli, Antonio, et al. “ASIF: Coupled Data Turns Unimodal Models to Multimodal without Training.” 37th Conference on Neural Information Processing Systems, vol. 36, Neural Information Processing Systems Foundation, 2023, pp. 15303–19. Norelli, A., Fumero, M., Maiorca, V., Moschella, L., Rodolà, E., & Locatello, F. (2023). ASIF: Coupled data turns unimodal models to multimodal without training. In 37th Conference on Neural Information Processing Systems (Vol. 36, pp. 15303–15319). New Orleans, LA, United States: Neural Information Processing Systems Foundation. 142162023-08-22T14:22:04Z2025-05-14T11:28:52Z