| Challenge: | Social media posts often contain unstructured text or images, making opinion mining challenging. |
| Approach: | They propose a new task for multimodal social media captions with named entities annotated and linked to external knowledge bases. |
| Outcome: | The proposed model outperforms state-of-the-art text-only NED models . it predicts correct entities in knowledge graph embeddings space, showing its efficacy and potentials . |
Similar Papers
Multimodal Named Entity Recognition for Short Social Media Posts (N18-1)
Copied to clipboard
| Challenge: | Social media posts often contain inconsistent or incomplete syntax and lexical notations with limited textual contexts. |
| Approach: | They propose a task called Multimodal Named Entity Recognition (MNER) for noisy user-generated data . they use a dataset called SnapCaptions to build upon the state-of-the-art NER models . |
| Outcome: | The proposed model outperforms existing models on noisy user-generated data . it uses a deep image network and generic modality attention module . |
Multimodal Knowledge Learning for Named Entity Disambiguation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing attempts to model multimodal information at the knowledge level are lacking multimodal annotation data against the large-scale unlabeled corpus. |
| Approach: | They propose to use multimodal knowledge learning to link ambiguous mentions with textual and visual contexts to a predefined knowledge graph. |
| Outcome: | The proposed method achieves improvements over the state-of-the-art methods on two public MNED datasets. |
Visual Attention Model for Name Tagging in Multimodal Social Media (P18-1)
Copied to clipboard
| Challenge: | Name tagging is a key task for language understanding, but is often limited by the short textual components. |
| Approach: | They propose a novel model architecture based on visual attention that outperforms other methods . they use multimodal datasets to analyze the name tagging task on social media . |
| Outcome: | The proposed model outperforms existing methods and significantly outperformed existing methods. |
Building a Multimodal Entity Linking Dataset From Tweets (2020.lrec-1)
Copied to clipboard
| Challenge: | Entity linking is a task that aims at associating an entity mention with a unique entity in a knowledge base. |
| Approach: | They propose a method to quasi-automatically build annotated datasets to evaluate methods on the Entity Linking task. |
| Outcome: | The proposed method builds annotated datasets of tweets with ambiguous mentions and a Twitter KB defining the entities. |
Grounded Multimodal Named Entity Recognition on Social Media (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies on Multimodal Named Entity Recognition only extract entity-type pairs in text, which is useless for multimodal knowledge graph construction. |
| Approach: | They propose a task to identify named entities in text and their bounding box groundings in image . they extend four well-known MNER methods to establish a number of baseline systems . |
| Outcome: | The proposed framework outperforms baseline systems on the GMNER task. |
In-context Learning for Few-shot Multimodal Named Entity Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for named entity recognition are time-consuming and laborintensive. |
| Approach: | They propose a few-shot multimodal named entity recognition task that uses few examples to locate and identify named entities for a text-image pair. |
| Outcome: | The proposed framework outperforms baselines under several few-shot settings. |
MNER-MI: A Multi-image Dataset for Multimodal Named Entity Recognition in Social Media (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent research has focused on multimodal named entity recognition (MNER) but current approaches focus on text and a single accompanying image, leaving a significant research gap in multi-image scenarios. |
| Approach: | They propose to construct a human-annotated MNER dataset with multiple images called MNER-MI and a temporal prompt model with multiple image to address the new challenges in multi-image scenarios. |
| Outcome: | The proposed method achieves state-of-the-art results on both MNER-MI and MNER -MI-Plus, demonstrating its effectiveness. |
Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for named entity recognition ignore visual context bias . NER is a key component of many information extraction tasks . |
| Approach: | They propose to use a multimodal interaction module to generate word-aware visual representations and leverage purely text-based entity span detection as an auxiliary module to guide the final predictions. |
| Outcome: | The proposed approach achieves state-of-the-art on two benchmark datasets. |
MELOV: Multimodal Entity Linking with Optimized Visual Features in Latent Space (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to multimodal entity linking focus on textual contexts but lack in social media vision modality. |
| Approach: | They propose a latent space vision feature optimization framework MELOV to address these challenges . they exploit variational autoencoder to mine shared information and generate text-based visual features . |
| Outcome: | The proposed framework is superior to existing methods on three benchmark datasets. |
Ameli: Enhancing Multimodal Entity Linking with Fine-Grained Attributes (2024.eacl-long)
Copied to clipboard
| Challenge: | Experimental results show that understanding attributes of mentions from text descriptions and visual images plays a vital role in multimodal entity linking. |
| Approach: | They propose to integrate attributes into multimodal entity linking using a text-image-based knowledge base. |
| Outcome: | The proposed approach integrates attributes into disambiguation. |