MCIL: Multimodal Counterfactual Instance Learning for Low-resource Entity-based Multimodal Information Extraction (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to perform multimodal information extraction only investigated entity-based tasks under supervised learning with adequate labeled data. |
| Approach: | They propose to investigate the entity-based MIE tasks under the low-resource settings by decomposing the features into image, entity, and context factors. |
| Outcome: | The proposed method is able to perform on two public MIE benchmark datasets and the experimental results confirm it. |
Similar Papers
MRE-MI: A Multi-image Dataset for Multimodal Relation Extraction in Social Media Posts (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to Multimodal Relation Extraction focus on single image scenarios . current approaches focus on text paired with a single image, ignoring valuable insights provided by remaining images. |
| Approach: | They propose a human-annotated dataset that includes multi-image and single-image instances for relation extraction. |
| Outcome: | The proposed model significantly improves relation extraction in multi-image scenarios. |
ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos (2023.emnlp-main)
Copied to clipboard
Te-Lin Wu, Zi-Yi Dou, Qingyuan Hu, Yu Hou, Nischal Chandra, Marjorie Freedman, Ralph Weischedel, Nanyun Peng
| Challenge: | despite its importance, there are few datasets that cover multimodal counterfactual reasoning . a dataset focusing on this area is limited because of its limited coverage over synthetic environments . |
| Approach: | They develop a video question answering dataset that provides questions on multimodal reasoning . they ask questions about counterfactual hypotheses over visual events . |
| Outcome: | The proposed dataset shows a significant performance gap between models and humans . it provides questions that span physical, social, and temporal dimensions . |
Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis (2023.acl-short)
Copied to clipboard
| Challenge: | Existing retrieval-augmented approaches focus on modeling the retrieved textual knowledge but this may not be able to accurately identify complex relations. |
| Approach: | They propose to retrieve multimodal relation extraction information based on object, sentence, and whole image . they propose to synthesize the object-level, image-level and sentence-level information . |
| Outcome: | The proposed method outperforms state-of-the-art models on multimodal relation extraction. |
Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View (2023.acl-long)
Copied to clipboard
| Challenge: | Special attention is paid to the cross-modal misalignment in text-image datasets which may mislead the learning. |
| Approach: | They propose a multimodal back-translation method which uses diffusion-based generative models for pseudo-paralleled pairs and a divergence estimator to construct a high-resource corpora as a bridge for low-ressource learners. |
| Outcome: | The proposed method outperforms 14 state-of-the-art methods in both entity and relation extraction tasks. |
Counterfactual Generator: A Weakly-Supervised Method for Named Entity Recognition (2020.emnlp-main)
Copied to clipboard
| Challenge: | Using labeled data, named entity recognition is labor-intensive, time-consuming and expensive. |
| Approach: | They propose a method which decomposes named entity into two parts: entity and context. |
| Outcome: | The proposed method improves the generalization ability of models under limited observational examples. |
Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) often rely on spurious correlations, undermining their robustness and generalization. |
| Approach: | They propose a causal mediation-based debiasing framework to address correlation bias in MLLMs . they distinguish core semantics from spurious textual and visual contexts using counterfactual examples . |
| Outcome: | The proposed framework surpasses existing state-of-the-art models on sarcasm detection and sentiment analysis tasks. |
Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling (2023.acl-long)
Copied to clipboard
| Challenge: | Existing research on multimodal relation extraction (MRE) faces internal-information over-utilization and external-information under-exploitation. |
| Approach: | They propose a framework that implements internal-information screening and external-information exploiting to address these challenges. |
| Outcome: | The proposed framework outperforms the current best model on the benchmark dataset. |
MNER-MI: A Multi-image Dataset for Multimodal Named Entity Recognition in Social Media (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent research has focused on multimodal named entity recognition (MNER) but current approaches focus on text and a single accompanying image, leaving a significant research gap in multi-image scenarios. |
| Approach: | They propose to construct a human-annotated MNER dataset with multiple images called MNER-MI and a temporal prompt model with multiple image to address the new challenges in multi-image scenarios. |
| Outcome: | The proposed method achieves state-of-the-art results on both MNER-MI and MNER -MI-Plus, demonstrating its effectiveness. |
OCR or Not? Rethinking Document Information Extraction in the MLLMs Era with Real-World Large-Scale Datasets (2026.eacl-industry)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) are used for document information extraction, but their impact on document information processing remains unclear. |
| Approach: | They propose an automated hierarchical error analysis framework that leverages large language models to diagnose errors systematically. |
| Outcome: | The proposed framework can achieve comparable performance to OCR-enhanced approaches. |
ImaRA: An Imaginative Frame Augmented Method for Low-Resource Multimodal Metaphor Detection and Explanation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for multimodal metaphor detection neglect cross-domain and attribute similarity characteristics underlying multimodal understanding. |
| Approach: | They propose an Imaginative FRame Augmented method for multimodal metaphor detection and explanation . they use a cross-modal imagination dataset rich in multimodal multimodal expressions . |
| Outcome: | The proposed method outperforms existing methods with training data on two datasets. |