Recognizing multimodal entailment

conference paper Recognizing multimodal entailment published yes Cesar Ilharco author Afsaneh Shirazi author Arjun Gopalan author Arsha Nagrani author Blaž Bratanič author Chris Bregler author Christina Liu author Felipe Ferreira author Gabriek Barcik author Gabriel Ilharco author Georg F Osang author 464B40D6-F248-11E8-B48F-1D18A9856A87 Jannis Bulian author Jared Frank author Lucas Smaira author Qin Cao author Ricardo Marino author Roma Patel author Thomas Leung author Vaiva Imbrasaite author HeEd department ACL: Association for Computational Linguistics ; IJCNLP: International Joint Conference on Natural Language Processing How information is created, shared and consumed has changed rapidly in recent decades, in part thanks to new social platforms and technologies on the web. With ever-larger amounts of unstructured and limited labels, organizing and reconciling information from different sources and modalities is a central challenge in machine learning. This cutting-edge tutorial aims to introduce the multimodal entailment task, which can be useful for detecting semantic alignments when a single modality alone does not suffice for a whole content understanding. Starting with a brief overview of natural language processing, computer vision, structured data and neural graph learning, we lay the foundations for the multimodal sections to follow. We then discuss recent multimodal learning literature covering visual, audio and language streams, and explore case studies focusing on tasks which require fine-grained understanding of visual and linguistic semantics question answering, veracity and hatred classification. Finally, we introduce a new dataset for recognizing multimodal entailment, exploring it in a hands-on collaborative section. Overall, this tutorial gives an overview of multimodal learning, introduces a multimodal entailment dataset, and encourages future research in the topic. https://research-explorer.ista.ac.at/download/10367/10368/2021_ACL_Ilharco.pdf application/pdfno https://creativecommons.org/licenses/by/4.0/ Association for Computational Linguistics2021Bangkok, Thailand eng 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Tutorial Abstracts 9-781-9540-8557-210.18653/v1/2021.acl-tutorials.6 29-30 C. Ilharco, A. Shirazi, A. Gopalan, A. Nagrani, B. Bratanič, C. Bregler, C. Liu, F. Ferreira, G. Barcik, G. Ilharco, G.F. Osang, J. Bulian, J. Frank, L. Smaira, Q. Cao, R. Marino, R. Patel, T. Leung, V. Imbrasaite, in:, 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Tutorial Abstracts, Association for Computational Linguistics, 2021, pp. 29–30. Ilharco, C., Shirazi, A., Gopalan, A., Nagrani, A., Bratanič, B., Bregler, C., … Imbrasaite, V. (2021). Recognizing multimodal entailment. In 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Tutorial Abstracts (pp. 29–30). Bangkok, Thailand: Association for Computational Linguistics. <a href="https://doi.org/10.18653/v1/2021.acl-tutorials.6">https://doi.org/10.18653/v1/2021.acl-tutorials.6</a> Ilharco C, Shirazi A, Gopalan A, et al. Recognizing multimodal entailment. In: 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Tutorial Abstracts. Association for Computational Linguistics; 2021:29-30. doi:<a href="https://doi.org/10.18653/v1/2021.acl-tutorials.6">10.18653/v1/2021.acl-tutorials.6</a> Ilharco, Cesar, Afsaneh Shirazi, Arjun Gopalan, Arsha Nagrani, Blaž Bratanič, Chris Bregler, Christina Liu, et al. “Recognizing Multimodal Entailment.” In 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Tutorial Abstracts, 29–30. Association for Computational Linguistics, 2021. <a href="https://doi.org/10.18653/v1/2021.acl-tutorials.6">https://doi.org/10.18653/v1/2021.acl-tutorials.6</a>. Ilharco C, Shirazi A, Gopalan A, Nagrani A, Bratanič B, Bregler C, Liu C, Ferreira F, Barcik G, Ilharco G, Osang GF, Bulian J, Frank J, Smaira L, Cao Q, Marino R, Patel R, Leung T, Imbrasaite V. 2021. Recognizing multimodal entailment. 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Tutorial Abstracts. ACL: Association for Computational Linguistics ; IJCNLP: International Joint Conference on Natural Language Processing, 29–30. Ilharco, Cesar, et al. “Recognizing Multimodal Entailment.” 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Tutorial Abstracts, Association for Computational Linguistics, 2021, pp. 29–30, doi:<a href="https://doi.org/10.18653/v1/2021.acl-tutorials.6">10.18653/v1/2021.acl-tutorials.6</a>. C. Ilharco et al., “Recognizing multimodal entailment,” in 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Tutorial Abstracts, Bangkok, Thailand, 2021, pp. 29–30. 103672021-11-28T23:01:30Z2022-01-26T14:26:36Z