Different Data, Different Modalities! Reinforced Data Splitting for Effective Multimodal Information Extraction from Social Media Posts (2022.coling-1)
Copied to clipboard
| Challenge: | Recent multimodal information extraction approaches overestimate the significance of images. |
| Approach: | They propose a general data splitting strategy to divide social media posts into two sets to achieve better performance under information extraction models of the corresponding modalities. |
| Outcome: | The proposed method outperforms existing models on two different multimodal information extraction tasks. |
Similar Papers
Improving Multimodal Classification of Social Media Posts by Leveraging Image-Text Auxiliary Tasks (2024.findings-eacl)
Copied to clipboard
| Challenge: | Prior work on multimodal content classification has not addressed these challenges. |
| Approach: | They propose to use two auxiliary tasks to fine-tune multimodal models to address hidden cross-modal semantics and weak image-text relationships when modeling text and images. |
| Outcome: | The proposed model improves by up to 2.6 F1 score across five diverse social media datasets. |
Bridging Modality Gap for Effective Multimodal Sentiment Analysis in Fashion-related Social Media (2025.coling-main)
Copied to clipboard
| Challenge: | Existing sentiment analysis tasks focus on text comprehension, but visual content is important for emotional expression. |
| Approach: | They propose a multimodal framework that integrates information from various modalities for sentiment classification of fashion posts. |
| Outcome: | The proposed framework outperforms existing unimodal and multimodal baselines on a comprehensive dataset and significantly outperformed existing unilmodal and multiple modal frameworks. |
MRE-MI: A Multi-image Dataset for Multimodal Relation Extraction in Social Media Posts (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to Multimodal Relation Extraction focus on single image scenarios . current approaches focus on text paired with a single image, ignoring valuable insights provided by remaining images. |
| Approach: | They propose a human-annotated dataset that includes multi-image and single-image instances for relation extraction. |
| Outcome: | The proposed model significantly improves relation extraction in multi-image scenarios. |
Towards Exploiting Sticker for Multimodal Sentiment Analysis in Social Media: A New Dataset and Baseline (2022.coling-1)
Copied to clipboard
| Challenge: | Sentiment analysis in social media is challenging because of the lack of context. |
| Approach: | They propose to use stickers to perform a multimodal sentiment analysis task using Chinese stickers. |
| Outcome: | The proposed model performs best compared with other models. |
Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View (2023.acl-long)
Copied to clipboard
| Challenge: | Special attention is paid to the cross-modal misalignment in text-image datasets which may mislead the learning. |
| Approach: | They propose a multimodal back-translation method which uses diffusion-based generative models for pseudo-paralleled pairs and a divergence estimator to construct a high-resource corpora as a bridge for low-ressource learners. |
| Outcome: | The proposed method outperforms 14 state-of-the-art methods in both entity and relation extraction tasks. |
Standardizing Distress Analysis: Emotion-Driven Distress Identification and Cause Extraction (DICE) in Multimodal Online Posts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for identifying hate speech have been limited to analyzing textual content. |
| Approach: | They propose a method for distress identification and cause extraction from social media posts using emotional information. |
| Outcome: | The proposed method improves F1 and ROS scores by 1.95% and 3% relative to the best-performing baseline. |
Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classification (2022.emnlp-main)
Copied to clipboard
| Challenge: | Social media users are using images and text to voice opinions and share ideas. |
| Approach: | They propose to use user comments to extract hinting features from user comments and explore them via self-training. |
| Outcome: | The proposed framework improves on four social media benchmarks for image-text relation classification, sarcasm detection, sentiment classification, and hate speech detection. |
Tackling Modality Heterogeneity with Multi-View Calibration Network for Multimodal Sentiment Detection (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies focus on fusing different features but ignore the challenge of modality heterogeneity. |
| Approach: | They propose a text-guided fusion module with novel Sparse-Attention to reduce the negative impacts of redundant visual elements and a sentiment-based congruity constraint task to calibrate the feature shift in the representation space. |
| Outcome: | The proposed model is competitive against existing methods and achieves state-of-the-art results on two public benchmark datasets. |
Adaptive Fusion Techniques for Multimodal Data (2021.eacl-main)
Copied to clipboard
| Challenge: | Effective fusion of data from multiple modalities is challenging due to the heterogeneous nature of multimodal data. |
| Approach: | They propose two adaptive fusion techniques that aim to combine multimodal data effectively. |
| Outcome: | The proposed networks can model context from other modalities better than existing methods. |
Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling (2023.acl-long)
Copied to clipboard
| Challenge: | Existing research on multimodal relation extraction (MRE) faces internal-information over-utilization and external-information under-exploitation. |
| Approach: | They propose a framework that implements internal-information screening and external-information exploiting to address these challenges. |
| Outcome: | The proposed framework outperforms the current best model on the benchmark dataset. |