| Challenge: | FrameNet Semantic Role Labeling aims to disambiguate situations around predicates using textual representations. |
| Approach: | They extend a frame identification task to leverage multimodal representations to improve FrameNet Semantic Role Labeling. |
| Outcome: | The proposed system outperforms its unimodal counterpart on the English frameNet and its German counterpart on IMAGINED words. |
Similar Papers
Do LLMs Encode Frame Semantics? Evidence from Frame Identification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using the FrameNet lexical resource, we evaluate large language models under prompt-based inference and observe that they can perform frame identification effectively even without explicit supervision. |
| Approach: | They evaluate large language models under prompt-based inference and observe that they encode latent knowledge of frame semantics. |
| Outcome: | The proposed model can generate coherent frame definitions while generalizing well to out-of-domain benchmarks. |
Combining ELECTRA and Adaptive Graph Encoding for Frame Identification (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies focus on FI tasks, but none have been done on the computational side. |
| Approach: | They propose a new system for Frame Identification based on pre-trained text encoders trained discriminatively and graphs embedding. |
| Outcome: | The proposed system produces state-of-the-art performance over two benchmarks and all possible splits and cleaning procedures used in the literature. |
Frame2: A FrameNet-based Multimodal Dataset for Tackling Text-image Interactions in Video (2024.lrec-main)
Copied to clipboard
Frederico Belcavello, Tiago Timponi Torrent, Ely E. Matos, Adriana S. Pagano, Maucha Gamonal, Natalia Sigiliano, Lívia Vicente Dutra, Helen de Andrade Abreu, Mairon Samagaio, Mariane Carvalho, Franciany Campos, Gabrielly Azalim, Bruna Mazzei, Mateus Fonseca de Oliveira, Ana Carolina Loçasso Luz, Lívia Pádua Ruiz, Júlia Bellei, Amanda Pestana, Josiane Costa, Iasmin Rabelo, Anna Beatriz Silva, Raquel Roza, Mariana Souza, Igor Oliveira
| Challenge: | et al., 2016) describe a multimodal dataset built from a Brazilian travel TV show . frameNet is composed of frames and their associated roles in a network of typed frame-to-frame relations. |
| Approach: | They present a multimodal dataset built from a Brazilian travel TV show annotated for FrameNet categories for both text and image communicative modes. |
| Outcome: | The proposed dataset includes 230 minutes of video annotated for FrameNet categories . the model can be applied to other communicative modes, i.e., images . |
Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View (2023.acl-long)
Copied to clipboard
| Challenge: | Special attention is paid to the cross-modal misalignment in text-image datasets which may mislead the learning. |
| Approach: | They propose a multimodal back-translation method which uses diffusion-based generative models for pseudo-paralleled pairs and a divergence estimator to construct a high-resource corpora as a bridge for low-ressource learners. |
| Outcome: | The proposed method outperforms 14 state-of-the-art methods in both entity and relation extraction tasks. |
Frame Semantics across Languages: Towards a Multilingual FrameNet (C18-3)
Copied to clipboard
| Challenge: | This workshop will present current research on aligning Frame Semantic resources across languages . resources based on FrameNet have been created for roughly a dozen languages based upon Fillmore's Frame Sementics . |
| Approach: | This workshop will present current research on aligning Frame Semantic resources across languages . resources based on FrameNet have been created for roughly a dozen languages based upon Fillmore's Frame Sementics . |
| Outcome: | This workshop will present current research on aligning Frame Semantic resources across languages and automatic frame semantic parsing in English and other languages. |
Framed Multi30K: A Frame-Based Multimodal-Multilingual Dataset (2024.lrec-main)
Copied to clipboard
Marcelo Viridiano, Arthur Lorenzi, Tiago Timponi Torrent, Ely E. Matos, Adriana S. Pagano, Natália Sathler Sigiliano, Maucha Gamonal, Helen de Andrade Abreu, Lívia Vicente Dutra, Mairon Samagaio, Mariane Carvalho, Franciany Campos, Gabrielly Azalim, Bruna Mazzei, Mateus Fonseca de Oliveira, Ana Carolina Luz, Livia Padua Ruiz, Júlia Bellei, Amanda Pestana, Josiane Costa, Iasmin Rabelo, Anna Beatriz Silva, Raquel Roza, Mariana Souza Mota, Igor Oliveira, Márcio Henrique Pelegrino de Freitas
| Challenge: | Recent advances in image-captioning datasets combine image and language to solve a diverse range of tasks. |
| Approach: | They propose a Brazilian Portuguese multimodal-multilingual dataset that extends the Multi30K dataset with 158,915 original Brazilian Portuguese descriptions and 30,104 Brazilian Portuguese translations. |
| Outcome: | The proposed dataset adds 2,677,613 frame evocation labels to the 158,915 English descriptions and to the ones created for Brazilian Portuguese. |
Aligning Images and Text with Semantic Role Labels for Fine-Grained Cross-Modal Understanding (2022.lrec-1)
Copied to clipboard
| Challenge: | Currently, image retrieval systems can retrieve relevant results for diverse inputs, but they do not provide a way to intentionally inject variety into the search results. |
| Approach: | They propose a multimodal dataset that combines semantic annotations with image bounding boxes. |
| Outcome: | The proposed system improves image retrieval performance and flexibility. |
Recognizing Multimodal Entailment (2021.acl-tutorials)
Copied to clipboard
Cesar Ilharco, Afsaneh Shirazi, Arjun Gopalan, Arsha Nagrani, Blaz Bratanic, Chris Bregler, Christina Funk, Felipe Ferreira, Gabriel Barcik, Gabriel Ilharco, Georg Osang, Jannis Bulian, Jared Frank, Lucas Smaira, Qin Cao, Ricardo Marino, Roma Patel, Thomas Leung, Vaiva Imbrasaite
| Challenge: | This tutorial introduces the multimodal entailment task for detecting semantic alignments . the task requires fine-grained understanding of visual and linguistic semantics questions . |
| Approach: | This tutorial introduces the multimodal entailment task to machine learning . it introduces a dataset for recognizing multimodal alignments . |
| Outcome: | This tutorial introduces the multimodal entailment task . it can be useful for detecting semantic alignments when a single modality alone is not enough . |
Crowdsourcing in the Development of a Multilingual FrameNet: A Case Study of Korean FrameNet (2020.lrec-1)
Copied to clipboard
| Challenge: | Using current methods, the construction of multilingual FrameNets is expensive and complex. |
| Approach: | They evaluated whether crowdsourcing approaches captured cross-cultural and cross-linguistic meanings . they found that crowd workers made intuitive choices comparable to trained FrameNet experts . |
| Outcome: | The results are now available in Korean FrameNet 1.1. |
Exploiting Definitions for Frame Identification (2021.eacl-main)
Copied to clipboard
| Challenge: | a frame-semantic parsing task is to determine which frame best captures the meaning of a word or phrase in a sentence. |
| Approach: | They propose a frame identification model that generates representations for frames and lexical units (senses) they evaluate the model on three data sets and show it consistently achieves better performance than previous systems. |
| Outcome: | The proposed model consistently outperforms previous systems on three data sets. |