Papers by Jiayi Fu
SocialGaze: Improving the Integration of Human Social Norms in Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Increasingly, large language models (LLMs) are able to understand and rationalize socially acceptable behaviors, but they are often misaligned with human consensus. |
| Approach: | They propose a multi-step prompting framework that verbalizes a social situation from multiple perspectives before forming a judgment. |
| Outcome: | The proposed framework improves the alignment with human judgments by up to 11 F1 points with the GPT-3.5 model. |
Reference Attack: A New Cross-Modal Jailbreaking Attack against Multimodal Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have raised significant safety concerns about generated content, drawing attention from both academia and industry. |
| Approach: | They propose a reference-guided cross-modal jailbreak method that enhances existing prompt-to-image injection attacks by exploiting MLLMs’ semantic reconstruction capabilities. |
| Outcome: | The proposed method achieves an attack success rate of over 93% on leading MLLMs including ChatGPT, Gemini, Claude, and the widely used open-source LLaMA model. |
“I See What You Did There”: Can Large Vision-Language Models Understand Multimodal Puns? (2026.acl-long)
Copied to clipboard
Naen Xu, Jiayi Sheng, Changjiang Li, Chunyi Zhou, Yuyuan Li, Tianyu Du, Jun Wang, Zhihui Fu, Jinbao Li, Shouling Ji
| Challenge: | Puns are a common form of rhetorical wordplay that exploits polysemy and phonetic similarity to create humor. |
| Approach: | They propose a multimodal pun generation pipeline and a model to evaluate their understanding of puns. |
| Outcome: | The proposed benchmark improves the understanding of multimodal puns by 16.5% in the F1 test. |
P4: Plug-and-Play Discrete Prompting for Large Language Models Personalization (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit impressive capabilities in following instructions, but manually prompting them to exhibit certain personalities may result in sub-optimal performance. |
| Approach: | They propose a plug-and-play prompting method to manipulate Large Language Models with distinct human-like personality traits by appending discrete personalized suffixes to query or dialog histories and focusing exclusively on influential tokens. |
| Outcome: | The proposed method outperforms other prompting methods and model editing methods on four models ranging from 1.1B to 13B and achieves 79.9% accuracy in customizing LLMs’ personalities. |
Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarios (2024.findings-acl)
Copied to clipboard
Lei Lin, Jiayi Fu, Pengli Liu, Qingyang Li, Yan Gong, Junchen Wan, Fuzheng Zhang, Zhongyuan Wang, Di Zhang, Kun Gai
| Challenge: | chain-of-thought (CoT) prompting has been shown to be effective on complex reasoning tasks, but the naive greedy decoding used in CoT prompting causes the repetitiveness and local optimality. |
| Approach: | They propose a generalizable ensemble-optimization method that uses a set of reasoning paths to prompt a language model one more time to determine the optimal answer. |
| Outcome: | The proposed method can be generalized to almost all scenarios where the type of input questions and answer format of reasoning paths may be unknown. |
Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement Learning with Verifiable Rewards (RLVR) has shown significant promise for enhancing the reasoning capabilities of large language models (LLMs). |
| Approach: | They propose a model-free method that refines credit assignment by leveraging the model's internal uncertainty signals. |
| Outcome: | Extensive experiments on five mathematical reasoning benchmarks show that the proposed method outperforms strong RLVR baselines on multiple model scales, including 1.5B and 7B. |
GumbelSoft: Diversified Language Model Watermarking via the GumbelMax-trick (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models generate human-like content, but they also pose a problem with generation diversity, negatively impacting generation diversity and user experience. |
| Approach: | They propose a Logits-Addition watermark and three variants that aim to enhance diversity to overcome generation diversity challenges. |
| Outcome: | The Logits-Addition watermark outperforms the Logits+Trick-based watermark in diversity tests and outperformed other decoding-based methods by 0.1 to 0.3. |