Enhancing Multimodal Named Entity Recognition through Adaptive Mixup Image Augmentation (2025.coling-main)
Copied to clipboard
| Challenge: | Current named entity recognition methods struggle with text-image mismatch problem due to a lack of visual context. |
| Approach: | They propose an adaptive mixup image augmentation method that generates augmented images based on matching score between text and image . |
| Outcome: | The proposed method can be integrated into existing models and demonstrate consistent performance improvements. |
Similar Papers
MNER-MI: A Multi-image Dataset for Multimodal Named Entity Recognition in Social Media (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent research has focused on multimodal named entity recognition (MNER) but current approaches focus on text and a single accompanying image, leaving a significant research gap in multi-image scenarios. |
| Approach: | They propose to construct a human-annotated MNER dataset with multiple images called MNER-MI and a temporal prompt model with multiple image to address the new challenges in multi-image scenarios. |
| Outcome: | The proposed method achieves state-of-the-art results on both MNER-MI and MNER -MI-Plus, demonstrating its effectiveness. |
Multimodal Named Entity Recognition for Short Social Media Posts (N18-1)
Copied to clipboard
| Challenge: | Social media posts often contain inconsistent or incomplete syntax and lexical notations with limited textual contexts. |
| Approach: | They propose a task called Multimodal Named Entity Recognition (MNER) for noisy user-generated data . they use a dataset called SnapCaptions to build upon the state-of-the-art NER models . |
| Outcome: | The proposed model outperforms existing models on noisy user-generated data . it uses a deep image network and generic modality attention module . |
Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for named entity recognition ignore visual context bias . NER is a key component of many information extraction tasks . |
| Approach: | They propose to use a multimodal interaction module to generate word-aware visual representations and leverage purely text-based entity span detection as an auxiliary module to guide the final predictions. |
| Outcome: | The proposed approach achieves state-of-the-art on two benchmark datasets. |
A Span-based Multimodal Variational Autoencoder for Semi-supervised Multimodal Named Entity Recognition (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for named entity recognition on social media are not efficient for semi-supervised MNER because of the mismatch between the posted text and image. |
| Approach: | They propose a novel method to fuse the text and image features for multimodal named entity recognition under semi-supervised setting by exploiting modal-specific VAEs. |
| Outcome: | The proposed method outperforms baselines under supervised setting and improves performance with less labeled data than existing semi-supervised methods. |
Flat Multi-modal Interaction Transformer for Named Entity Recognition (2022.coling-1)
Copied to clipboard
| Challenge: | Named entity recognition (MNER) aims at identifying entity spans and recognizing their categories in social media posts with the aid of images. |
| Approach: | They propose to use sentences and general domain words to obtain visual cues to transform the fine-grained semantic representation of vision and text into a unified lattice structure and leverage entity boundary detection as an auxiliary task to alleviate visual bias. |
| Outcome: | The proposed method achieves state-of-the-art on two benchmark datasets. |
Memory-Guided Hard Data Augmentation for Multimodal Named Entity Recognition (2026.findings-acl)
Copied to clipboard
Xinyu Liu, Kai fu, Yinghan Shi, Quanyou Chu, Ming Du, Hongya Wang, Xiaojun Meng, Jiansheng Wei, Yanghua Xiao, Bo Xu
| Challenge: | Existing methods for Named Entity Recognition (NER) ignore the internal state of the target model. |
| Approach: | They propose a framework to repair model-specific errors by using a model-based approach . they employ cross-validation to identify model- specific Hard Data and a memory tree to induce macro-level error patterns from micro-level failures. |
| Outcome: | The proposed framework yields significant performance gains on Twitter and other platforms. |
Grounded Multimodal Named Entity Recognition on Social Media (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies on Multimodal Named Entity Recognition only extract entity-type pairs in text, which is useless for multimodal knowledge graph construction. |
| Approach: | They propose a task to identify named entities in text and their bounding box groundings in image . they extend four well-known MNER methods to establish a number of baseline systems . |
| Outcome: | The proposed framework outperforms baseline systems on the GMNER task. |
Boosting Textural NER with Synthetic Image and Instructive Alignment (2024.findings-acl)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a key task reliant on textual data. |
| Approach: | They propose a method to transform NER into a multimodal task by using images from the internet as auxiliaries. |
| Outcome: | The proposed method surpasses all text-only baselines and improves F1 score by 1.4% to 2.3% on prominent MNER datasets. |
Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with Auxiliary Refined Knowledge (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to enhance textual entity prediction neglect the need for external knowledge or encounter high redundancy in the retrieved knowledge. |
| Approach: | They propose a framework that leverages ChatGPT as an implicit knowledge base and heuristically generates auxiliary knowledge for more efficient entity prediction. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on two classic datasets and exhibits a stronger robustness and generalization capability. |
An Effective Span-based Multimodal Named Entity Recognition with Consistent Cross-Modal Alignment (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to name entity recognition rely on word-based sequence labeling and align image and text at inconsistent semantic levels. |
| Approach: | They propose a span-based method which achieves a more consistent multimodal alignment from the perspectives of information-theoretic and cross-modal interaction. |
| Outcome: | Experiments on two datasets show that SMNER outperforms the state-of-the-art methods. |