Papers by Jinghong Lin
Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Multimodal Models (LMMs) have shown promise in hateful meme detection, but they face limitations like sub-optimal performance and limited out-of-domain generalization capabilities. |
| Approach: | They propose a robust adaptation framework for hateful meme detection that enhances in-domain accuracy and cross-domain generalization while preserving the general vision-language capabilities of LMMs. |
| Outcome: | The proposed framework outperforms larger agentic systems in detecting hateful memes under adversarial attacks while maintaining the general vision-language capabilities of LMMs. |
Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning (2024.acl-long)
Copied to clipboard
| Challenge: | Existing systems for detecting hateful memes lack sensitivity to subtle differences in memes that are vital for correct hatefulness classification. |
| Approach: | They propose to construct a hatefulness-aware embedding space through retrieval-guided contrastive training to identify hatefulness based on data unseen in training. |
| Outcome: | The proposed system outperforms existing models on the HatefulMemes dataset with an AUROC of 87.0 and improves contextual understanding across domains. |
Retrieval-Augmented Defense: Adaptive and Controllable Jailbreak Prevention for Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) remain vulnerable to jailbreak attacks due to evolving nature and diversity of attack strategies. |
| Approach: | They propose a framework for jailbreak detection that integrates a database of known attack examples into Retrieval-Augmented Generation to infer the underlying, malicious user query and jailbreak strategy used to attack the system. |
| Outcome: | The proposed framework reduces the effectiveness of strong jailbreak attacks while maintaining low rejection rates for benign queries. |
PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers (2024.acl-long)
Copied to clipboard
| Challenge: | Large Multimodal Models excel in natural language and visual understanding but are challenged by challenging tasks such as Knowledge-based Visual Question Answering (KB-VQA). |
| Approach: | They propose a framework for training Large Multimodal Models (LMMs) to perform KB-VQA tasks. |
| Outcome: | The proposed framework is used to train and evaluate multi-modal retrievers. |
Control-DAG: Constrained Decoding for Non-Autoregressive Directed Acyclic T5 using Weighted Finite State Automata (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing non-autoregressive (NAR) models fail to generate specified entity names in up to 40% of responses and produce OOV errors. |
| Approach: | They propose a constrained decoding algorithm for Directed Acyclic T5 model which offers lexical, vocabulary and length control. |
| Outcome: | The proposed model significantly improves on Schema Guided Dialogue and DART datasets, establishing strong results for Task-Oriented Dialog and Data-to-Text NLG. |
Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding (2024.naacl-short)
Copied to clipboard
| Challenge: | Recent work shows that MBR decoding can significantly improve translation performance of Multilingual Large Language Models. |
| Approach: | They propose a method that uses a monolingual fine-tuning set to fine- tune MLLMs to get the gains of MBR without additional computation in inference. |
| Outcome: | The proposed method outperforms greedy decoding and beam search on multiple NMT tests. |
Matching Article Pairs with Graphical Decomposition and Convolutions (P19-1)
Copied to clipboard
| Challenge: | Existing methods for matching sentence pairs do not perform well in longer documents . Existing approaches for matching sentences do not work in longer document understanding tasks . |
| Approach: | They propose to model article pairs by comparing sentences that enclose same concept vertex . they propose to use a concept interaction graph to match articles by encoding sentences . |
| Outcome: | The proposed methods show significant improvements over existing methods . the proposed datasets consist of 30K pairs of breaking news articles . |