Papers by Haolong Yan
Leveraging Generative Large Language Models with Visual Instruction and Demonstration Retrieval for Multimodal Sarcasm Detection (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for multimodal sarcasm detection do not fully utilize cross-modal features, limiting their performance on in-domain datasets. |
| Approach: | They propose a multimodal sarcasm detection model with a designed instruction template and a demonstration retrieval module. |
| Outcome: | The proposed model outperforms existing methods on in-domain datasets and achieves state-of-the-art performance. |
Visual Enhanced Entity-Level Interaction Network for Multimodal Summarization (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to generate concise summarizations rely on coarse-grained textual and visual information, but they are underutilized. |
| Approach: | They propose a Visual Enhanced Entity-Level Interaction Network to address underutilization of multimodal inputs at a fine-grained level. |
| Outcome: | The proposed model outperforms existing models on two MMS datasets and proposes new metrics to measure factual consistency of entities in the output. |
Entity-level Interaction via Heterogeneous Graph for Multimodal Named Entity Recognition (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for name-based entity recognition neglect the integrity of entity semantics and conduct cross-modal interaction at token-level. |
| Approach: | They propose a multimodal named entity recognition model that captures visual information and fuses it into tokens to rid non-entity tokens of visual noise. |
| Outcome: | The proposed model captures entity-related visual information and fuses it into tokens . it eliminates visual noise and makes non-entity tokens easily misidentified as entities . |