Latent space translation via semantic alignment

Maiorca V, Moschella L, Norelli A, Fumero M, Locatello F, Rodolà E. Latent space translation via semantic alignment. arXiv, 2311.00664.

Download (ext.)

Preprint | Submitted | English
Author
Maiorca, Valentino; Moschella, Luca; Norelli, Antonio; Fumero, Marco; Locatello, FrancescoISTA ; Rodolà, Emanuele
Department
Abstract
While different neural models often exhibit latent spaces that are alike when exposed to semantically related data, this intrinsic similarity is not always immediately discernible. Towards a better understanding of this phenomenon, our work shows how representations learned from these neural modules can be translated between different pre-trained networks via simpler transformations than previously thought. An advantage of this approach is the ability to estimate these transformations using standard, well-understood algebraic procedures that have closed-form solutions. Our method directly estimates a transformation between two given latent spaces, thereby enabling effective stitching of encoders and decoders without additional training. We extensively validate the adaptability of this translation procedure in different experimental settings: across various trainings, domains, architectures (e.g., ResNet, CNN, ViT), and in multiple downstream tasks (classification, reconstruction). Notably, we show how it is possible to zero-shot stitch text encoders and vision decoders, or vice-versa, yielding surprisingly good classification performance in this multimodal setting.
Publishing Year
Date Published
2023-11-01
Journal Title
arXiv
Acknowledgement
This work is supported by the ERC grant no.802554 (SPECGEO), PRIN 2020 project no.2020TA3K9N (LEGO.AI), and PNRR MUR project PE0000013-FAIR. Francesco Locatello did not contribute to this work at Amazon.
Article Number
2311.00664
IST-REx-ID

Cite this

Maiorca V, Moschella L, Norelli A, Fumero M, Locatello F, Rodolà E. Latent space translation via semantic alignment. arXiv. doi:10.48550/arXiv.2311.00664
Maiorca, V., Moschella, L., Norelli, A., Fumero, M., Locatello, F., & Rodolà, E. (n.d.). Latent space translation via semantic alignment. arXiv. https://doi.org/10.48550/arXiv.2311.00664
Maiorca, Valentino, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodolà. “Latent Space Translation via Semantic Alignment.” ArXiv, n.d. https://doi.org/10.48550/arXiv.2311.00664.
V. Maiorca, L. Moschella, A. Norelli, M. Fumero, F. Locatello, and E. Rodolà, “Latent space translation via semantic alignment,” arXiv. .
Maiorca V, Moschella L, Norelli A, Fumero M, Locatello F, Rodolà E. Latent space translation via semantic alignment. arXiv, 2311.00664.
Maiorca, Valentino, et al. “Latent Space Translation via Semantic Alignment.” ArXiv, 2311.00664, doi:10.48550/arXiv.2311.00664.
All files available under the following license(s):
Copyright Statement:
This Item is protected by copyright and/or related rights. [...]

Link(s) to Main File(s)
Access Level
OA Open Access

Export

Marked Publications

Open Data ISTA Research Explorer

Sources

arXiv 2311.00664

Search this title in

Google Scholar