Papers by Xiang Yue

Differential Privacy for Text Analytics via Natural Text Sanitization (2021.findings-acl)

Challenge:	a small learning gap exists between large and small language models . long CoT data and large model responses are not beneficial for small models - a problem that may be due to the small student model's ability to handle distribution shifts.
Approach:	They propose a mix distillation strategy that balances reasoning complexity by combining long and short CoT examples or reasoning from both larger and smaller models.
Outcome:	The proposed strategy outperforms training on large and small models on short CoT and small model CoT.

Copied to clipboard

Xiang Yue, Minxin Du, Tianhao Wang, Yaliang Li, Huan Sun, Sherman S. M. Chow

Semantic Text Processing

Challenge:	Existing text sanitization mechanisms provide low utility, as cursed by the high-dimensional text representation.
Approach:	They propose to use sanitized texts to samaritize training data . they propose to retrain and fine-tune the senitization-aware language model .
Outcome:	The proposed approach enables privacypreserving natural language processing over the BERT language model with promising utility.

AgentDiagnose: An Open Toolkit for Diagnosing LLM Agent Trajectories (2025.emnlp-demos)

Copied to clipboard

Tianyue Ou, Wanyao Guo, Apurva Gandhi, Graham Neubig, Xiang Yue

Challenge:	Large Language Model (LLM) agents produce rich, multi-step trajectories that interleave observations, internal reasoning, and tool actions.
Approach:	They propose an open-source framework for diagnosing agent trajectories that quantifies five core agentic competencies and a visualization module that highlights trajectory semantics.
Outcome:	The proposed framework is extensible and compatible with most agent trajectories.

C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References (2022.acl-short)

Copied to clipboard

Xiang Yue, Xiaoman Pan, Wenlin Yao, Dian Yu, Dong Yu, Jianshu Chen

Semantic Text Processing

Challenge:	Existing approaches to pretrain open-domain question answering systems lack task-specific annotations.
Approach:	They propose to pretrain a two-stage open-domain question answering system with strong transfer capabilities by using a dictionary and a large-scale corpus.
Outcome:	The proposed approach leads to 2%-10% gains in top-20 accuracy and improves with reader.

Evaluating Language Models as Synthetic Data Generators (2025.acl-long)

Copied to clipboard

Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence, Sean Welleck, Graham Neubig

Scaling Evaluation-Time Compute with Reasoning Models as Evaluators (2026.findings-acl)

Challenge:	Prior studies have focused on developing effective data generation methods, but lack systematic comparison of different LMs as data generators in a unified setting.
Approach:	They propose to use a benchmark to compare language models' data generation abilities against a set of standardized settings and metrics.
Outcome:	The proposed benchmark provides standardized settings and metrics to evaluate LMs’ data generation abilities.

Copied to clipboard

Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Minkyeong Moon, Carolin Lawrence, Kiril Gashteovski, Julia Hockenmaier, Graham Neubig, Sean Welleck

Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA (2025.findings-acl)

Challenge:	Language model (LM) evaluators that generate chain-of-thought reasoning are widely used for the assessment of LM responses.
Approach:	They investigate whether increasing LMs' "thinking" time through scaling test-time compute can improve an LM's evaluation capability.
Outcome:	The proposed reasoning models improve evaluation performance monotonically with the number of reasoning tokens generated, mirroring trends seen in LM reasoning.

Copied to clipboard

Qianqi Yan, Xuehai He, Xiang Yue, Xin Eric Wang

Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe (2023.acl-long)

Challenge:	Large Multimodal Models (LMMs) have demonstrated impressive performance on existing medical visual question answering benchmarks.
Approach:	They evaluate large multimodal models that perform worse than random guessing on medical questions . authors suggest more robust evaluation methods to ensure reliability of LMMs .
Outcome:	a new study shows that large multimodal models perform worse than random guessing on medical visual question answering benchmarks.

Copied to clipboard

Xiang Yue, Huseyin Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David Levitan, Robert Sim

Ethical NLP

Challenge:	Privacy concerns have increased in data-driven products due to the tendency of machine learning models to memorize sensitive training data.
Approach:	They propose a method for generating useful synthetic text with a formal privacy guarantee by fine-tuning a pretrained generative language model with DP.
Outcome:	The proposed method produces synthetic text competitive in terms of utility with its non-private counterpart, while providing strong protection against potential privacy leakages.

Rethinking the Evaluation of In-Context Learning for LLMs (2024.emnlp-main)

Copied to clipboard

Guoxin Yu, Lemao Liu, Mo Yu, Yue Yu, Xiang Ao

Semantic Text Processing

Challenge:	Existing studies evaluate In-context learning methods based on task performance . however, this evaluation protocol overlooks the significant cost associated with the demonstration configuration process .
Approach:	They propose a two-dimensional evaluation paradigm that considers both configuration costs and task performance.
Outcome:	The proposed evaluation paradigm can be applied to any ICL method as a plugin.

From Selection to Generation: A Survey of LLM-based Active Learning (2025.acl-long)

Copied to clipboard

Yu Xia, Subhojyoti Mukherjee, Zhouhang Xie, Junda Wu, Xintong Li, Ryan Aponte, Hanjia Lyu, Joe Barrow, Hongjie Chen, Franck Dernoncourt, Branislav Kveton, Tong Yu, Ruiyi Zhang, Jiuxiang Gu, Nesreen K. Ahmed, Yu Wang, Xiang Chen, Hanieh Deilamsalehy, Sungchul Kim, Zhengmian Hu, Yue Zhao, Nedim Lipka, Seunghyun Yoon, Ting-Hao Kenneth Huang, Zichao Wang, Puneet Mathur, Soumyabrata Pal, Koyel Mukherjee, Zhehao Zhang, Namyong Park, Thien Huu Nguyen, Jiebo Luo, Ryan A. Rossi, Julian McAuley

Video-MMMU: Evaluating Knowledge Acquisition from Multidisciplinary Professional Videos (2026.acl-long)

Challenge:	Large Language Models (LLMs) have been used for selection and training of data for active learning.
Approach:	They propose an intuitive taxonomy that categorizes LLM-based active learning techniques and discuss the transformative roles they can play in the active learning loop.
Outcome:	The proposed model can generate entirely new data instances and provide more cost-effective annotations with fewer labeled data instances.

Copied to clipboard

Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Xiang Yue, Bo Li, Yuanhan Zhang, Ziwei Liu

Visual Data in NLP

Challenge:	Existing video benchmarks do not evaluate the knowledge acquisition capabilities of Large Multimodal Models (LMMs) existing video benchmark focuses on static, general visual understanding tasks, without evaluating whether models can acquire knowledge dynamically.
Approach:	They propose a multi-modal, multi-discipline, multitrack benchmark that evaluates Large Multimodal Models’ ability to acquire knowledge from college-level, educational videos.
Outcome:	The proposed benchmark reveals a substantial gap between human learners and current Large Multimodal Models (LMMs) and focuses on improving their learning efficiency.

AttributionBench: How Hard is Automatic Attribution Evaluation? (2024.findings-acl)

Copied to clipboard

Yifei Li, Xiang Yue, Zeyi Liao, Huan Sun

Challenge:	generative search engines enhance the reliability of large language model responses by providing cited evidence.
Approach:	They propose to use a benchmark to evaluate whether a large language model supports the generated responses or not .
Outcome:	The proposed benchmark shows that even a fine-tuned GPT-3.5 only achieves around 80% macro-F1 under a binary classification formulation.

MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale (2025.acl-long)

Copied to clipboard

Jiawei Guo, Tianyu Zheng, Yizhi Li, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Graham Neubig, Wenhu Chen, Xiang Yue

ReSel: N-ary Relation Extraction from Scientific Text and Tables by Learning to Retrieve and Select (2022.emnlp-main)

Challenge:	Current instruction-tuning datasets focus on simplistic visual question answering tasks, and provide phrase-level answers without any intermediate rationales.
Approach:	They propose to use open-source multimodal large language models to train MLLMs on a dataset with 12M instruction-response pairs to elicit CoT reasoning.
Outcome:	The proposed model achieves state-of-the-art performance on benchmarks such as MathVerse, MMMU-Pro, and MuirBench, and gains improvements of up to 4% on non-reasoning-based benchmarks.

Copied to clipboard

Yuchen Zhuang, Yinghao Li, Junyang Zhang, Yue Yu, Yingjun Mou, Xiang Chen, Le Song, Chao Zhang

Information Extraction & Text Mining

Challenge:	Our proposed method extracts N-ary relation tuples from scientific articles.
Approach:	They propose a method that decomposes the task into two stages . they propose modal query and modal entity selection . their results show that ReSel outperforms state-of-the-art baselines significantly .
Outcome:	The proposed method outperforms state-of-the-art baselines on three scientific information extraction datasets.

MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2025.acl-long)

Copied to clipboard

Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, Graham Neubig

WIND: Weighting Instances Differentially for Model-Agnostic Domain Adaptation (2021.findings-acl)

Challenge:	Recent advances in multimodal large language models have led to progress in tackling complex reasoning tasks that combine textual and visual information.
Approach:	They introduce a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark.
Outcome:	The proposed model performs lower on MMMU-Pro than on the previous benchmark, ranging from 16.8% to 26.9%.

Copied to clipboard

Xiang Chen, Yue Cao, Xiaojun Wan

Low-Resource NLP

Challenge:	Existing methods for instance weighting cannot learn the weights which make the model generalize well in target domain.
Approach:	They propose a modelagnostic instance weighting algorithm which can learn the instance weights instead of manually designed weighting metrics.
Outcome:	The proposed method can learn the instance weights instead of manually designed weighting metrics.

COUGH: A Challenge Dataset and Models for COVID-19 FAQ Retrieval (2021.emnlp-main)

Copied to clipboard

Xinliang Frederick Zhang, Heming Sun, Xiang Yue, Simon Lin, Huan Sun

Information Retrieval

Challenge:	16K FAQ items scraped from 55 credible websites . 32 human-annotated FAQ items for each query.
Approach:	They present a large, challenging dataset for FAQ retrieval for COVID-19 . they use a FAQ bank, Query Bank and Relevance Set to evaluate the dataset .
Outcome:	The proposed model achieves 48.8 under P@5 and is compared with other datasets.

Machine Unlearning of Pre-trained Large Language Models (2024.acl-long)

Copied to clipboard

Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, Xiang Yue

VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search (2025.emnlp-main)

Challenge:	Using curated datasets, we establish a robust benchmark for unlearning performance, demonstrating that these methods are over 105 times more computationally efficient than retraining.
Approach:	They propose a framework for machine unlearning in pre-trained LLMs and integrate gradient ascent with gradient descent on in-distribution data to achieve robustness.
Outcome:	The proposed framework is over 105 times more efficient than retraining on in-distribution data and provides detailed guidelines for efficient hyperparameter tuning in the unlearning process.

Copied to clipboard

Yiming Jia, Jiachen Li, Xiang Yue, Bo Li, Ping Nie, Kai Zou, Wenhu Chen

VisCoder: Fine-Tuning LLMs for Executable Python Visualization Code Generation (2025.findings-emnlp)

Challenge:	Existing vision-language models struggle with reasoning-focused tasks due to the lack of high-quality training data.
Approach:	They propose a new approach that leverages search engines to create a multimodal multimodal dataset . they use a set of 30,000 seed images to extract HTML data from 700K unique URLs .
Outcome:	The proposed model achieves the best known performance on MMMU-Pro (40.7), MathVerse (42.6), and DynaMath (55.7).

Copied to clipboard

Yuansheng Ni, Ping Nie, Kai Zou, Xiang Yue, Wenhu Chen

Temporal Sampling for Forgotten Reasoning in LLMs (2026.acl-long)

Challenge:	Existing instruction-tuning datasets lack execution-grounded supervision and offer limited support for iterative code correction.
Approach:	They propose a large-scale instruction tuning dataset for Python-based visualization and self-correction.
Outcome:	The proposed dataset outperforms strong open-source baselines and proprietary models like GPT-4o-mini.

Copied to clipboard

Yuetai Li, Zhangchen Xu, Fengqing Jiang, Bhaskar Ramasubramanian, Luyao Niu, Bill Yuchen Lin, Xiang Yue, Radha Poovendran

VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation (2024.acl-long)

Challenge:	a new metric measures the percentage of questions that were answered incorrectly during fine-tuning .
Approach:	They propose a decoding strategy that draws outputs from multiple checkpoints along the training trajectory.
Outcome:	The proposed method improves reasoning performance and consistency across benchmarks.

Copied to clipboard

Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, Wenhu Chen

Explainability & Interpretability in NLP

Challenge:	Existing metrics for conditional image generation are opaque and lack explainability . evaluators of these metrics have limited ability to evaluate image synthesis tasks .
Approach:	They propose a Visual Instruction-guided Explainable metric for evaluating conditional image models.
Outcome:	The proposed model achieves a high Spearman correlation with human evaluations, but is weaker than GPT-4o and GPT-v in evaluating synthetic images.

Automatic Evaluation of Attribution by Large Language Models (2023.findings-emnlp)

Copied to clipboard

Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, Huan Sun

Unveiling Privacy Risks in LLM Agent Memory (2025.acl-long)

Challenge:	Generative large language models (LLMs) incorporate external references to generate and support claims. however, evaluating the attribution remains an open problem.
Approach:	They investigate automatic evaluation of attribution given by large language models . they define different types of attributed errors and then explore two approaches .
Outcome:	The proposed methods highlight promising signals and challenges.

Copied to clipboard

Bo Wang, Weiyi He, Shenglai Zeng, Zhen Xiang, Yue Xing, Jiliang Tang, Pengfei He

Flexibly Utilize Memory for Long-Term Conversation via a Fragment-then-Compose Framework (2025.emnlp-main)

Challenge:	Large Language Model (LLM) agents store private user-agent interactions in memory for demonstrations, introducing new privacy risks for LLM agents.
Approach:	They propose an attack that extracts private information from memory under a black-box setting and propose a method that can be used to attack the agent.
Outcome:	The proposed attack is effective under a black-box setting and it is demonstrated on two representative agents.

Copied to clipboard

Cai Ke, Yiming Du, Bin Liang, Yifan Xiang, Lin Gui, Zhongyang Li, Baojun Wang, Yue Yu, Hui Wang, Kam-Fai Wong, Ruifeng Xu

Dialogue Systems & Conversational Agents

Challenge:	Large language models extract useful information from conversation history to enhance the response in long-term conversations.
Approach:	They propose a Fragment-then-Compose framework to optimize memory utilization for long-term open-domain conversation.
Outcome:	The proposed framework can be used to extract useful information from conversation history . it can be adapted to different situations and improve response generation .

Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate (2023.findings-emnlp)

Copied to clipboard

Boshi Wang, Xiang Yue, Huan Sun

JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation (2025.naacl-long)

Challenge:	Large language models (LLMs) have shown impressive performance in complex reasoning tasks, but it is difficult to know whether they are reasoning based on deep understandings of truth and logic or leveraging their vast previously-seen patterns in a relatively shallow way.
Approach:	They propose to test large language models by engaging with them in a debate-like conversation where the user and LLM need to discuss to make the correct decision starting from opposing arguments.
Outcome:	The proposed model can achieve the correct answer on its own, but can also hold and defend its belief instead of blindly believing or getting misled by the user’s (invalid) arguments and critiques.

Copied to clipboard

Shota Onohara, Atsuyuki Miyai, Yuki Imajuku, Kazuki Egashira, Jeonghun Baek, Xiang Yue, Graham Neubig, Kiyoharu Aizawa

Synthetic Question Value Estimation for Domain Adaptation of Question Answering (2022.acl-long)

Challenge:	Using culture-agnostic subsets, performance drops in many LMMs when evaluated in Japanese.
Approach:	They introduce a Japanese benchmark to evaluate large multimodal models on expert-level tasks based on the Japanese cultural context.
Outcome:	The proposed benchmark enables comparisons with other benchmarks in other languages based on cultural contexts.

Copied to clipboard

Xiang Yue, Ziyu Yao, Huan Sun

TableLlama: Towards Open Large Generalist Models for Tables (2024.naacl-long)

Challenge:	Existing work adapts QA scores to select high-quality questions, but these scores do not improve QA performance on the target domain.
Approach:	They propose to synthesize QA pairs with a question generator on the target domain . they propose to train a Question Value Estimator that estimates usefulness of synthetic questions .
Outcome:	The proposed method improves the performance of the target domain QA model by using synthetic questions and only 15% of the human annotations on the targetdomain.

Copied to clipboard

Tianshu Zhang, Xiang Yue, Yifei Li, Huan Sun

SimulBench: Evaluating Language Models with Creative Simulation Tasks (2025.findings-naacl)

Challenge:	Existing methods for interpreting, augmenting, and querying semi-structured tables require pretraining on tables or special model architecture design.
Approach:	They construct a dataset with a variety of tables and tasks for instruction tuning and evaluating LLMs.
Outcome:	The proposed model achieves comparable or better performance on 7 out of 8 in-domain tasks compared with the base model on 6 out-of-domain datasets.

Copied to clipboard

Qi Jia, Xiang Yue, Tuney Zheng, Jie Huang, Bill Yuchen Lin

Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset (2020.acl-main)

Challenge:	Existing benchmarks for large language models do not fully evaluate their potential for broad implementation.
Approach:	They propose to use a fixed LLM as a user agent to engage with an LLM to collect dialogues first under different tasks.
Outcome:	The proposed framework outperforms LLaMA-3-70b-Chat on 18.55% more cases.

Copied to clipboard

Xiang Yue, Bernal Jimenez Gutierrez, Huan Sun

Machine Reading Comprehension

Challenge:	Medical professionals often query over clinical notes to find information that can support their decision making.
Approach:	They propose to use expert-annotated question templates and existing i2b2 annotations to create emrQA, the first large-scale dataset for question answering based on clinical notes.
Outcome:	The proposed system can answer clinical questions without using domain knowledge.

LIME: Less Is More for MLLM Evaluation (2025.findings-acl)

Copied to clipboard

King Zhu, Qianbo Zang, Shian Jia, Siwei Wu, Feiteng Fang, Yizhi Li, Shuyue Guo, Tianyu Zheng, Jiawei Guo, Bo Li, Haoning Wu, Xingwei Qu, Jian Yang, Ruibo Liu, Xiang Yue, Jiaheng Liu, Chenghua Lin, Hamid Alinejad-Rokny, Min Yang, Shiwen Ni, Wenhao Huang, Ge Zhang