ASIF: Coupled data turns unimodal models to multimodal without training Advances in Neural Information Processing Systems Norelli, Antonio Fumero, Marco Maiorca, Valentino Moschella, Luca Rodolà, Emanuele Locatello, Francesco ; https://orcid.org/0000-0002-4850-0683 ddc:000 CLIP proved that aligning visual and language spaces is key to solving many vision tasks without explicit training, but required to train image and text encoders from scratch on a huge dataset. LiT improved this by only training the text encoder and using a pre-trained vision network. In this paper, we show that a common space can be created without any training at all, using single-domain encoders (trained with or without supervision) and a much smaller amount of image-text pairs. Furthermore, our model has unique properties. Most notably, deploying a new version with updated training samples can be done in a matter of seconds. Additionally, the representations in the common space are easily interpretable as every dimension corresponds to the similarity of the input to a unique entry in the multimodal dataset. Experiments on standard zero-shot visual benchmarks demonstrate the typical transfer ability of image-text models. Overall, our method represents a simple yet surprisingly strong baseline for foundation multi-modal models, raising important questions on their data efficiency and on the role of retrieval in machine learning. Neural Information Processing Systems Foundation 2023 info:eu-repo/semantics/conferenceObject doc-type:conferenceObject text https://research-explorer.ista.ac.at/record/14216 https://research-explorer.ista.ac.at/download/14216/18994 Norelli A, Fumero M, Maiorca V, Moschella L, Rodolà E, Locatello F. ASIF: Coupled data turns unimodal models to multimodal without training. In: <i>37th Conference on Neural Information Processing Systems</i>. Vol 36. Neural Information Processing Systems Foundation; 2023:15303-15319. eng info:eu-repo/semantics/altIdentifier/isbn/9781713899921 info:eu-repo/semantics/altIdentifier/arxiv/2210.01738 info:eu-repo/semantics/openAccess