| Challenge: | Social media posts often contain inconsistent or incomplete syntax and lexical notations with limited textual contexts. |
| Approach: | They propose a task called Multimodal Named Entity Recognition (MNER) for noisy user-generated data . they use a dataset called SnapCaptions to build upon the state-of-the-art NER models . |
| Outcome: | The proposed model outperforms existing models on noisy user-generated data . it uses a deep image network and generic modality attention module . |
Similar Papers
Multimodal Named Entity Disambiguation for Noisy Social Media Posts (P18-1)
Copied to clipboard
| Challenge: | Social media posts often contain unstructured text or images, making opinion mining challenging. |
| Approach: | They propose a new task for multimodal social media captions with named entities annotated and linked to external knowledge bases. |
| Outcome: | The proposed model outperforms state-of-the-art text-only NED models . it predicts correct entities in knowledge graph embeddings space, showing its efficacy and potentials . |
MNER-MI: A Multi-image Dataset for Multimodal Named Entity Recognition in Social Media (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent research has focused on multimodal named entity recognition (MNER) but current approaches focus on text and a single accompanying image, leaving a significant research gap in multi-image scenarios. |
| Approach: | They propose to construct a human-annotated MNER dataset with multiple images called MNER-MI and a temporal prompt model with multiple image to address the new challenges in multi-image scenarios. |
| Outcome: | The proposed method achieves state-of-the-art results on both MNER-MI and MNER -MI-Plus, demonstrating its effectiveness. |
Grounded Multimodal Named Entity Recognition on Social Media (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies on Multimodal Named Entity Recognition only extract entity-type pairs in text, which is useless for multimodal knowledge graph construction. |
| Approach: | They propose a task to identify named entities in text and their bounding box groundings in image . they extend four well-known MNER methods to establish a number of baseline systems . |
| Outcome: | The proposed framework outperforms baseline systems on the GMNER task. |
In-context Learning for Few-shot Multimodal Named Entity Recognition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for named entity recognition are time-consuming and laborintensive. |
| Approach: | They propose a few-shot multimodal named entity recognition task that uses few examples to locate and identify named entities for a text-image pair. |
| Outcome: | The proposed framework outperforms baselines under several few-shot settings. |
A Span-based Multimodal Variational Autoencoder for Semi-supervised Multimodal Named Entity Recognition (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for named entity recognition on social media are not efficient for semi-supervised MNER because of the mismatch between the posted text and image. |
| Approach: | They propose a novel method to fuse the text and image features for multimodal named entity recognition under semi-supervised setting by exploiting modal-specific VAEs. |
| Outcome: | The proposed method outperforms baselines under supervised setting and improves performance with less labeled data than existing semi-supervised methods. |
Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for named entity recognition ignore visual context bias . NER is a key component of many information extraction tasks . |
| Approach: | They propose to use a multimodal interaction module to generate word-aware visual representations and leverage purely text-based entity span detection as an auxiliary module to guide the final predictions. |
| Outcome: | The proposed approach achieves state-of-the-art on two benchmark datasets. |
Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts (2022.aacl-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a longstanding NLP task that consists of identifying an entity in a sentence or document. |
| Approach: | They construct a dataset of seven entity types annotated over 11,382 tweets . they provide a set of language model baselines and analyze the performance of the model . |
| Outcome: | The proposed dataset contains seven entity types annotated over 11,382 tweets . the authors focus on short-term degradation of NER models over time and strategies to fine-tune a language model over different periods . |
Visual Attention Model for Name Tagging in Multimodal Social Media (P18-1)
Copied to clipboard
| Challenge: | Name tagging is a key task for language understanding, but is often limited by the short textual components. |
| Approach: | They propose a novel model architecture based on visual attention that outperforms other methods . they use multimodal datasets to analyze the name tagging task on social media . |
| Outcome: | The proposed model outperforms existing methods and significantly outperformed existing methods. |
NERetrieve: Dataset for Next Generation Named Entity Recognition and Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a widely adopted NLP task . authors present three variants of NER task, with dataset to support them . |
| Approach: | They propose three variants of the NER task, together with a dataset to support them . they propose a move towards more fine-grained entities and zero-shot recognition . |
| Outcome: | The proposed model matches or surpasses existing models in NER tasks . the proposed model is based on a large, silver-annotated corpus of 4 million paragraphs . |
An Effective Span-based Multimodal Named Entity Recognition with Consistent Cross-Modal Alignment (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to name entity recognition rely on word-based sequence labeling and align image and text at inconsistent semantic levels. |
| Approach: | They propose a span-based method which achieves a more consistent multimodal alignment from the perspectives of information-theoretic and cross-modal interaction. |
| Outcome: | Experiments on two datasets show that SMNER outperforms the state-of-the-art methods. |