Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Multimodal Models (LMMs) have shown promise in hateful meme detection, but they face limitations like sub-optimal performance and limited out-of-domain generalization capabilities. |
| Approach: | They propose a robust adaptation framework for hateful meme detection that enhances in-domain accuracy and cross-domain generalization while preserving the general vision-language capabilities of LMMs. |
| Outcome: | The proposed framework outperforms larger agentic systems in detecting hateful memes under adversarial attacks while maintaining the general vision-language capabilities of LMMs. |
Similar Papers
A Context-Aware Contrastive Learning Framework for Hateful Meme Detection and Segmentation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Empirical experiments show HateSieve surpasses existing LMMs in performance with fewer trainable parameters . |
| Approach: | They propose a framework to enhance detection and segmentation of hateful elements in memes by creating a triplet dataset and an Image-Text Alignment module. |
| Outcome: | HateSieve features a new framework that creates semantically correlated memes and generates contextual embeddings for accurate meme segmentation. |
Caption Enriched Samples for Improving Hateful Memes Detection (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for classifying memes are difficult to perform, with human accuracy only about 85% . recent state-of-the-art models perform considerably less accurately, achieving up to 64.73% accuracy. |
| Approach: | They propose to use an off-the-shelf caption generator to capture the first image and overlayed text. |
| Outcome: | The proposed tool improves classification accuracy for unimodal and multimodal models . the proposed tool can be used to model the contrast between image content and overlayed text . |
Text or Image? What is More Important in Cross-Domain Generalization Capabilities of Hate Meme Detection Models? (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies show that only the textual component of hateful memes enables the multimodal classifier to generalize across domains while the image component proves highly sensitive to a specific training dataset. |
| Approach: | They propose to use only the textual component of hateful memes to generalize across different domains while the image component is highly sensitive to a specific training dataset. |
| Outcome: | The proposed model performs similarly to hate-meme classifiers in a zero-shot setting, while the introduction of meme’s image captions worsens performance by an average F1 of 0.02. |
Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning (2024.acl-long)
Copied to clipboard
| Challenge: | Existing systems for detecting hateful memes lack sensitivity to subtle differences in memes that are vital for correct hatefulness classification. |
| Approach: | They propose to construct a hatefulness-aware embedding space through retrieval-guided contrastive training to identify hatefulness based on data unseen in training. |
| Outcome: | The proposed system outperforms existing models on the HatefulMemes dataset with an AUROC of 87.0 and improves contextual understanding across domains. |
AdamMeme: Adaptively Probe the Reasoning Capacity of Multimodal Large Language Models on Harmfulness (2025.acl-long)
Copied to clipboard
| Challenge: | Existing models that assess mLLMs on harmful meme understanding are inaccurate and lack accuracy. |
| Approach: | They propose a framework that adaptively probes the reasoning capabilities of mLLMs . their framework systematically reveals the varying performance of different target mllms a . |
| Outcome: | The proposed framework systematically reveals the performance of different target mLLMs. |
MemeIntel: Explainable Detection of Propagandistic and Hateful Memes (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for label detection and explanation generation have been limited in understanding complex issues . identifying propaganda and hate in memes is essential for combating misinformation and minimizing harm . |
| Approach: | They propose an explanation-enhanced dataset for propaganda memes in Arabic and hateful memes on English to solve these tasks. |
| Outcome: | The proposed model outperforms the current state-of-the-art in label detection and explanation generation. |
Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for harmful meme detection ignore in-depth cognition of meme text and image . authors propose a framework for learning reasonable thoughts from LLMs for better multimodal fusion . |
| Approach: | They propose to use large language models to learn reasonable thoughts from LLMs for better multimodal fusion and lightweight fine-tuning. |
| Outcome: | The proposed approach achieves superior performance than state-of-the-art methods on the harmful meme detection task. |
Deciphering Implicit Hate: Evaluating Automated Detection Algorithms for Multimodal Hate (2021.findings-acl)
Copied to clipboard
| Challenge: | Imlicit hate content has unusual syntax, polysemic words, and fewer markers of prejudice, e.g., slurs . multimodal content is harder to detect than unimodal content, such as memes . |
| Approach: | They evaluate the role of semantic and multimodal context for detecting implicit and explicit hate . they find that all models perform better on content with full annotator agreement . |
| Outcome: | The proposed model outperforms other models on implicit and explicit hate detection tasks because of its lower propensity towards false positives. |
Deciphering Hate: Identifying Hateful Memes and Their Targets (2024.acl-long)
Copied to clipboard
| Challenge: | a growing body of research has focused on the negative aspects of memes in high-resource languages like Bengali . a new dataset for Bengali hateful memes is designed to detect their targeted entities . |
| Approach: | They propose a multimodal dataset that analyzes the modality of memes and compares them with other datasets. |
| Outcome: | The proposed dataset outperforms state-of-the-art datasets on Bengali hateful memes . the proposed dataset is generalizable on other low-resource hateful memes datasets compared with baselines based on the proposed model . |
Evolver: Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for detecting hateful memes rely on extensive training. |
| Approach: | They propose a method that integrates evolution attribute and in-context information of memes into large multimodal models via Chain-of-Evolution (CoE) prompting. |
| Outcome: | The proposed method improves existing methods on public datasets and can be used as interpretive tool to promote understanding of evolution of memes. |