Scale Is All You Need: Analyzing Modality Interaction and Speaker Intent Without Fine-Tuning

Challenge:	Recent work on sarcasm and humor detection uses large multimodal Transformers, but they are computationally expensive and opaque.
Approach:	They propose a lightweight framework for multimodal sarcasm detection that combines frozen text, audio, and visual embeddings from pretrained encoders through compact fusion heads.
Outcome:	The proposed framework improves on the best unimodal baseline by combining text, audio, and visual embeddings from pretrained encoders with compact fusion heads.

Similar Papers

Overview

Challenge

Approach

Outcome

Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper) (P19-1)

Copied to clipboard

Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, Soujanya Poria

Sentiment Analysis

Challenge:	sarcasm is often expressed through multiple verbal and non-verbal cues, such as a change of tone, overemphasis, drawn-out syllables, or a straight looking face.
Approach:	They propose to use multimodal cues to improve sarcasm detection using audiovisual utterances annotated with sarcasm labels to improve the accuracy.
Outcome:	The proposed dataset reduces the error rate of sarcasm detection by 12.9% . it is based on audiovisual utterances annotated with sarcasm labels .

Leveraging Generative Large Language Models with Visual Instruction and Demonstration Retrieval for Multimodal Sarcasm Detection (2024.naacl-long)

Copied to clipboard

Binghao Tang, Boda Lin, Haolong Yan, Si Li

Multimodality

Challenge:	Existing methods for multimodal sarcasm detection do not fully utilize cross-modal features, limiting their performance on in-domain datasets.
Approach:	They propose a multimodal sarcasm detection model with a designed instruction template and a demonstration retrieval module.
Outcome:	The proposed model outperforms existing methods on in-domain datasets and achieves state-of-the-art performance.

A Multimodal Corpus for Emotion Recognition in Sarcasm (2022.lrec-1)

Copied to clipboard

Anupama Ray, Shubham Mishra, Apoorva Nunna, Pushpak Bhattacharyya

Sentiment Analysis

Challenge:	sarcasm and emotion are often used in conversational systems to generate the right response.
Approach:	They use a sarcastic expression dataset pre-annotated with 9 emotions to detect emotion . they identify and correct 343 incorrect emotion labels and label each sarkastic utterance with one of four sarcasm types.
Outcome:	The proposed model outperforms state-of-the-art sarcasm detection methods by using a multimodal sarcastic detection dataset.

CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models (2024.acl-long)

Copied to clipboard

Zixin Chen, Hongzhan Lin, Ziyang Luo, Mingfei Cheng, Jing Ma, Guang Chen

Sentiment Analysis

Challenge:	Current methods for multimodal sarcasm target identification focus on superficial indicators in an end-to-end manner, overlooking the nuanced understanding of multimodal content.
Approach:	They propose a multimodal sarcasm target identification framework with a coarse-to-fine paradigm by augmenting sarcasm explainability with reasoning and pre-training knowledge.
Outcome:	The proposed framework outperforms state-of-the-art methods and exhibits explainability in deciphering sarcasm as well.

When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues (2022.acl-long)

Copied to clipboard

Shivani Kumar, Atharva Kulkarni, Md Shad Akhtar, Tanmoy Chakraborty

Sentiment Analysis

Challenge:	Indirect speech achieves a constellation of discourse goals in human communication, but it is challenging for AI agents to comprehend such idiosyncrasies.
Approach:	They propose a task to generate natural language explanations of satirical conversations using a multimodal and code-mixed dataset to capture multimodality.
Outcome:	The proposed task generates natural language explanations of satirical conversations in a multimodal and code-mixed setting and surpasses baselines on almost all metrics.

Action and Reaction Go Hand in Hand! a Multi-modal Dialogue Act Aided Sarcasm Identification (2024.lrec-main)

Copied to clipboard

Mohit Singh Tomar, Tulika Saha, Abhisek Tiwari, Sriparna Saha

Natural Language Interfaces

Challenge:	Existing studies have shown that sarcasm is reflected by the intended meaning of the speaker's utterance.
Approach:	They propose to extend the MUStARD dataset to enclose dialogue acts for each dialogue . they propose a dialogue act-aided multi-modal transformer network for sarcasm identification model .
Outcome:	The proposed model improves performance in dialogue act-aided sarcasm identification compared to sardasmatic identification alone.

SarcNet: A Multilingual Multimodal Sarcasm Detection Dataset (2024.lrec-main)

Copied to clipboard

Tan Yue, Xuzhao Shi, Rui Mao, Zonghai Hu, Erik Cambria

Multimodality

Challenge:	Sarcasm is an implicit form of sarcasm, involving an intended meaning that contradicts the literal expression . human use conflict between factual information and a statement as cues to detect sarcasm . sarkasmatic analysis is challenging due to its implicit nature .
Approach:	They propose a multimodal sarcasm detection dataset that uses multiple modalities to detect sarcasm.
Outcome:	The proposed model improves on previous models based on a single label . human sarcasm cannot be detected using a unified label across multiple modalities .

Breakthrough from Nuance and Inconsistency: Enhancing Multimodal Sarcasm Detection with Context-Aware Self-Attention Fusion and Word Weight Calculation. (2024.lrec-main)

Copied to clipboard

Hongfei Xue, Linyan Xu, Yu Tong, Rui Li, Jiali Lin, Dazhi Jiang

Sentiment Analysis

Challenge:	Existing methods for sarcasm detection rely on feature concatenation to fuse different modalities or model inconsistencies among modalités.
Approach:	They propose to use Context-Aware Self-Attention Fusion to integrate local and momentary multimodal information into specific words to illustrate the inconsistencies between connotation and denotation.
Outcome:	The proposed method achieves an accuracy of 76.9 and an F1 score of 76.1 on the MUStARD dataset, surpassing the current state-of-the-art IWAN model by 1.7 and 1.6 respectively.

Ambiguity-aware Multi-level Incongruity Fusion Network for Multi-Modal Sarcasm Detection (2025.coling-main)

Copied to clipboard

Kuntao Li, Yifan Chen, Qiaofeng Wu, Weixing Mai, Fenghuan Li, Yun Xue

Multimodality

Challenge:	Existing methods for sarcasm detection focus on fusing text and image information to establish cross-modal correlations, overlooking the significance of original unimodal incongruity information.
Approach:	They propose a multi-modal incongruity learning module to capture inconcluity information simultaneously at the text-level, image-level and cross-modal-level.
Outcome:	The proposed model outperforms state-of-the-art methods on a publicly available dataset.

Predict and Use: Harnessing Predicted Gaze to Improve Multimodal Sarcasm Detection (2023.emnlp-main)

Copied to clipboard

Divyank Tiwari, Diptesh Kanojia, Anupama Ray, Apoorva Nunna, Pushpak Bhattacharyya

Sentiment Analysis

Challenge:	sarcasm detection depends on content spoken, tonality, facial expressions, context, and personal traits like language proficiency and cognitive capabilities.
Approach:	They propose to use synthetic gaze data to improve sarcasm detection in conversational context . they collect gaze features for 20% of data instances and use them to predict gaze features .
Outcome:	The proposed model improves performance on a conversational dataset using gaze features . it achieves a gain of 6.6% points on the complete dataset with only predicted gaze features.