Papers by Jerry Li
Investigating the Multilingual Calibration Effects of Language Model Instruction Tuning (2026.eacl-short)
Copied to clipboard
Jerry Huang, Peng Lu, Qiuhao Zeng, Yusuke Iwasawa, Yutaka Matsuo, Sarath Chandar, Edison Marrese-Taylor, Irene Li
| Challenge: | despite advances in foundation model research, the relationship between large language models and their calibration remains an open area of research. |
| Approach: | They examine a gap in the calibration of large language models within multilingual settings to better understand how data scarcity can potentially lead to different calibration effects. |
| Outcome: | The proposed calibration gap is found in two multilingual benchmarks over 29 and 42 languages. |
MemeQA: Holistic Evaluation for Meme Understanding (2025.acl-long)
Copied to clipboard
Khoi P. N. Nguyen, Terrence Li, Derek Lou Zhou, Gabriel Xiong, Pranav Balu, Nandhan Alahari, Alan Huang, Tanush Chauhan, Harshavardhan Bala, Emre Guzelordu, Affan Kashfi, Aaron Xu, Suyesh Shrestha, Megan Vu, Jerry Wang, Vincent Ng
| Challenge: | Existing benchmarks for meme understanding only concern narrow aspects of meme semantics. |
| Approach: | They propose to use multiple-choice questions to evaluate meme comprehension . they use a dataset of over 9,000 multiple-question questions to assess meme comprehension. |
| Outcome: | The proposed model outperforms existing models on meme comprehension . the model makes many errors on memes where proper understanding requires going beyond sentiment . |
STORM-BORN: A Challenging Mathematical Derivations Dataset Curated via a Human-in-the-Loop Multi-Agent Framework (2025.findings-acl)
Copied to clipboard
Wenhao Liu, Zhenyi Lu, Xinyu Hu, Jerry Zhang, Dailin Li, Jiacheng Cen, Huilin Cao, Haiteng Wang, Yuhan Li, Xie Kun, Dandan Li, Pei Zhang, Chengbo Zhang, Yuxiang Ren, Xiaohong Huang, Yan Ma
| Challenge: | Existing datasets suffer from outdated and insufficient challenging content, neglecting human-like reasoning, and limited reliability due to single-LLM generation. |
| Approach: | They propose a human-in-the-loop, multi-agent data generation framework that integrates reasoning-dense filters, multiagent collaboration, and human mathematicians’ evaluations to ensure the reliability and quality of the dataset. |
| Outcome: | The proposed framework improves accuracy and quality of the 2,000-synthesized datasets by integrating reasoning-dense filters, multi-agent collaboration, and human mathematicians’ evaluations. |
When Identity Skews Debate: Anonymization for Bias-Reduced Multi-Agent Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Multi-agent debate (MAD) aims to improve large language model reasoning by letting multiple agents exchange answers and then aggregate their opinions. |
| Approach: | They propose a principled framework that joins sycophancy and self-bias to mitigate and quantify identity bias in multi-agent debate by removing identity markers from prompts. |
| Outcome: | The proposed framework joins identity-driven sycophancy and self-bias to mitigate and quantify identity bias in multi-agent debate. |
MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application (2026.acl-long)
Copied to clipboard
Xueqing Peng, Lingfei Qian, Yan Wang, Ruoyu Xiang, Yueru He, Yang Ren, Mingyang Jiang, Vincent Jim Zhang, Yuqing Guo, Jeff Zhao, Huan He, Yi Han, Yun Feng, Yuechen Jiang, Yupeng Cao, Haohang Li, Yangyang Yu, Xiaoyu Wang, Penglei Gao, Shengyuan Lin, Keyi Wang, Shanshan Yang, Yilun Zhao, Zhiwei Liu, Peng Lu, Jerry Huang, Suyuchen Wang, Triantafillos Papadopoulos, Polydoros Giannouris, Efstathia Soufleri, Nuo Chen, Zhiyang Deng, Heming Fu, Yijia Zhao, Mingquan Lin, Meikang Qiu, Kaleb E Smith, Arman Cohan, Xiao-Yang Liu, Jimin Huang, Guojun Xiong, Alejandro Lopez-Lira, Xi Chen, Junichi Tsujii, Jian-Yun Nie, Sophia Ananiadou, Qianqian Xie
| Challenge: | Existing evaluations of LLMs in finance are text-only, monolingual, and largely saturated by current models. |
| Approach: | They propose a multilingual and multimodal benchmark for evaluating LLMs in real financial contexts. |
| Outcome: | The first expert-annotated multilingual and multimodal benchmark is released . it evaluates 21 leading LLMs and shows they perform better in multilingual settings . |
Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used as rerankers, but their ranking behavior can be steered by small, natural-sounding prompts. |
| Approach: | They propose a token optimization method that crafts concise textual perturbations to consistently promote a target item in LLM-generated rankings. |
| Outcome: | The proposed method outperforms state-of-the-art base-lines and is hard to detect. |
Automatic Prompt Optimization with “Gradient Descent” and Beam Search (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown impressive performance as general purpose agents, but their abilities remain highly dependent on prompts which are hand written with onerous trial-and-error effort. |
| Approach: | They propose an algorithm that uses numerical gradient descent to automatically improve prompts by rewriting vague task descriptions into more precise annotation instructions. |
| Outcome: | The proposed algorithm outperforms previous methods and improves performance on three benchmark NLP tasks and the novel problem of LLM jailbreak detection. |
POLYIE: A Dataset of Information Extraction from Polymer Material Scientific Literature (2024.naacl-long)
Copied to clipboard
Jerry Cheung, Yuchen Zhuang, Yinghao Li, Pranav Shetty, Wantian Zhao, Sanjeev Grampurohit, Rampi Ramprasad, Chao Zhang
| Challenge: | SciIE datasets for polymer materials are lacking for this class of materials . POLYIE is curated from 146 full-length polymer scholarly articles . |
| Approach: | They propose a SciIE dataset for polymer materials that uses entity annotations from 146 full-length articles. |
| Outcome: | The proposed dataset is curated from 146 full-length polymer scholarly articles . it presents challenges due to diverse lexical formats of entities and ambiguity between entities . |