Detector-free weakly supervised grounding by separation

Arbelle A, Doveh S, Alfassy A, Shtok J, Lev G, Schwartz E, Kuehne H, Levi HB, Sattigeri P, Panda R, Chen C-F, Bronstein AM, Saenko K, Ullman S, Giryes R, Feris R, Karlinsky L. 2021. Detector-free weakly supervised grounding by separation. IEEE/CVF International Conference on Computer Vision. ICCV: International Conference on Computer Vision vol. 15.

Download (ext.)

Conference Paper | Published | English

Scopus indexed
Author
Arbelle, Assaf; Doveh, Sivan; Alfassy, Amit; Shtok, Joseph; Lev, Guy; Schwartz, Eli; Kuehne, Hilde; Levi, Hila Barak; Sattigeri, Prasanna; Panda, Rameswar; Chen, Chun-Fu; Bronstein, Alex M.ISTA
All
Abstract
Nowadays, there is an abundance of data involving images and surrounding free-form text weakly corresponding to those images. Weakly Supervised phrase-Grounding (WSG) deals with the task of using this data to learn to localize (or to ground) arbitrary text phrases in images without any additional annotations. However, most recent SotA methods for WSG assume an existence of a pre-trained object detector, relying on it to produce the ROIs for localization. In this work, we focus on the task of Detector-Free WSG (DF-WSG) to solve WSG without relying on a pre-trained detector. The key idea behind our proposed Grounding by Separation (GbS) method is synthesizing ‘text to image-regions’ associations by random alpha-blending of arbitrary image pairs and using the corresponding texts of the pair as conditions to recover the alpha map from the blended image via a segmentation network. At test time, this allows using the query phrase as a condition for a non-blended query image, thus interpreting the test image as a composition of a region corresponding to the phrase and the complement region. Our GbS shows an 8.5% accuracy improvement over previous DF-WSG SotA, for a range of benchmarks including Flickr30K, Visual Genome, and ReferIt, as well as a complementary improvement (above 7%) over the detector-based approaches for WSG.
Publishing Year
Date Published
2021-10-20
Proceedings Title
IEEE/CVF International Conference on Computer Vision
Publisher
Institute of Electrical and Electronics Engineers
Volume
15
Conference
ICCV: International Conference on Computer Vision
Conference Location
Montreal, Canada
Conference Date
2021-10-10 – 2021-10-17
IST-REx-ID

Cite this

Arbelle A, Doveh S, Alfassy A, et al. Detector-free weakly supervised grounding by separation. In: IEEE/CVF International Conference on Computer Vision. Vol 15. Institute of Electrical and Electronics Engineers; 2021. doi:10.1109/iccv48922.2021.00182
Arbelle, A., Doveh, S., Alfassy, A., Shtok, J., Lev, G., Schwartz, E., … Karlinsky, L. (2021). Detector-free weakly supervised grounding by separation. In IEEE/CVF International Conference on Computer Vision (Vol. 15). Montreal, Canada: Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/iccv48922.2021.00182
Arbelle, Assaf, Sivan Doveh, Amit Alfassy, Joseph Shtok, Guy Lev, Eli Schwartz, Hilde Kuehne, et al. “Detector-Free Weakly Supervised Grounding by Separation.” In IEEE/CVF International Conference on Computer Vision, Vol. 15. Institute of Electrical and Electronics Engineers, 2021. https://doi.org/10.1109/iccv48922.2021.00182.
A. Arbelle et al., “Detector-free weakly supervised grounding by separation,” in IEEE/CVF International Conference on Computer Vision, Montreal, Canada, 2021, vol. 15.
Arbelle A, Doveh S, Alfassy A, Shtok J, Lev G, Schwartz E, Kuehne H, Levi HB, Sattigeri P, Panda R, Chen C-F, Bronstein AM, Saenko K, Ullman S, Giryes R, Feris R, Karlinsky L. 2021. Detector-free weakly supervised grounding by separation. IEEE/CVF International Conference on Computer Vision. ICCV: International Conference on Computer Vision vol. 15.
Arbelle, Assaf, et al. “Detector-Free Weakly Supervised Grounding by Separation.” IEEE/CVF International Conference on Computer Vision, vol. 15, Institute of Electrical and Electronics Engineers, 2021, doi:10.1109/iccv48922.2021.00182.
All files available under the following license(s):
Copyright Statement:
This Item is protected by copyright and/or related rights. [...]

Link(s) to Main File(s)
Access Level
OA Open Access

Export

Marked Publications

Open Data ISTA Research Explorer

Sources

arXiv 2104.09829

Search this title in

Google Scholar