Papers by Yixin Liu

BlindGuard: Safeguarding LLM-based Multi-Agent Systems under Unknown Attacks (2026.acl-long)

Challenge:	Language Models (LMs) have demonstrated impressive molecule understanding ability on 1D text-related tasks, but lack 2D graph perception, a critical ability of human professionals in comprehending molecules’ topological structures.
Approach:	They propose to combine a cross-modal projector and a uni-modal adapter to enable an LM to understand both text- and graph-based molecular contents via a Q-Former.
Outcome:	The proposed model outperforms the baselines on tasks such as molecule captioning, IUPAC name prediction, and molecule-text retrieval.

Copied to clipboard

Rui Miao, Yixin Liu, Yili Wang, Xu Shen, Yue Tan, Yiwei Dai, Shirui Pan, Xin Wang

Robustness in NLP

Challenge:	Existing supervised defense methods rely on labeled malicious agents to train a supervised model of malicious behavior.
Approach:	They propose an unsupervised defense method that learns without requiring any attack-specific labels or prior knowledge of malicious behaviors.
Outcome:	The proposed method detects diverse attack types across MAS with various communication patterns while maintaining superior generalizability compared to baselines.

LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing (2024.emnlp-main)

Copied to clipboard

Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Cheng Jiayang, Zhaowei Wang, Ying Su, Raj Shah, Ruohao Guo, Jing Gu, Haoran Li, Kangda Wei, Zihao Wang, Lu Cheng, Surangika Ranathunga, Meng Fang, Jie Fu, Fei Liu, Ruihong Huang, Eduardo Blanco, Yixin Cao, Rui Zhang, Philip Yu, Wenpeng Yin

Challenge:	a comparative analysis of paper (meta-)reviews by large language models (LLMs) aims to identify and distinguish LLMs from human activities .
Approach:	They present a comparative analysis to identify and distinguish LLM activities from human activities.
Outcome:	The proposed analysis aims to improve recognition of instances when someone implicitly uses LLMs for reviewing activities.

Joint Representation Learning of Cross-lingual Words and Entities via Attentive Distant Supervision (D18-1)

Copied to clipboard

Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Chengjiang Li, Xu Chen, Tiansi Dong

FOLIO: Natural Language Reasoning with First-Order Logic (2024.emnlp-main)

Challenge:	Existing methods for learning word and entity representations in monolingual settings are limited.
Approach:	They propose a method for joint representation learning of cross-lingual words and entities that captures mutually complementary knowledge and enables cross-linguistic inferences.
Outcome:	The proposed method captures mutually complementary knowledge and enables cross-lingual inferences among knowledge bases and texts.

Copied to clipboard

Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, Dragomir Radev

Where Fact Ends and Fairness Begins: Redefining AI Bias Evaluation through Cognitive Biases (2025.findings-emnlp)

Challenge:	Existing benchmarks for logical reasoning in large language models lack language naturalness or limited complexity.
Approach:	They propose to use first-order logic annotations to evaluate logical reasoning capabilities of large language models.
Outcome:	The proposed dataset evaluates the FOL reasoning ability of supervised fine-tuning on medium-sized language models.

Copied to clipboard

Jen-tse Huang, Yuhang Yan, Linqi Liu, Yixin Wan, Wenxuan Wang, Kai-Wei Chang, Michael R. Lyu

On Learning to Summarize with Large Language Models as References (2024.naacl-long)

Challenge:	Existing benchmarks conflate factual correctness and normative fairness . a model may generate responses that are factually accurate but socially unfair .
Approach:	They propose a benchmark to examine the boundary between fact and fair . they draw on representativeness bias, attribution bias and ingroup–outgroup bias to explain why models often misalign fact and faireness.
Outcome:	The proposed model is based on ten frontier models and is available on github . it is compared with a standard model that generates people of color in Nazi-era uniforms .

Copied to clipboard

Yixin Liu, Kejian Shi, Katherine He, Longtian Ye, Alexander Fabbri, Pengfei Liu, Dragomir Radev, Arman Cohan

MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues (2026.findings-acl)

Challenge:	Recent studies have found that summaries generated by large language models (LLMs) are favored by human annotators when compared to reference summary from widely used summarization datasets.
Approach:	They propose to use large language models (LLMs) as reference learning settings for smaller text summarization models to investigate whether their performance can be substantially improved.
Outcome:	The proposed model outperforms standard supervised fine-tuning and human evaluations while retaining human-level performance.

Copied to clipboard

Zheyuan Liu, Dongwhi Kim, Yixin Wan, Xiangchi Yuan, Zhaoxuan Tan, Fengran Mo, Meng Jiang

Dialogue Systems & Conversational Agents

Challenge:	Existing contextual safety benchmarks are mostly single-turn and miss how malicious intent can emerge gradually or how the same scene can support both benign and exploitative goals.
Approach:	They propose a benchmark that evaluates contextual safety in multimodal large language models . they observe persistent trade-offs between contextual safety and utility .
Outcome:	The proposed model combines multi-turn and multi-switch scenarios to evaluate safety in multimodal large language models.

PRISM: Probabilistic Reward Model with Inherent Structural Modeling (2026.acl-long)

Copied to clipboard

Yuhang Zhou, Yixin Cao, Yuchen Ni, Shihan Dou, Xutian Chen, Ge Zhang, Xiang Liu, Guangnan Ye

Cognitive Modeling

Challenge:	Existing evaluators compress diverse human judgments into a single scalar, leading to brittle alignment and reward hacking.
Approach:	They propose a Gaussian-based reinterpretation of reward evaluation as a conditional distribution and a mixture of Gaussians to capture conflicting preference dimensions.
Outcome:	The proposed model outperforms scalar baselines in accuracy and generalization.

ExplainaBoard: An Explainable Leaderboard for NLP (2021.acl-demo)

Copied to clipboard

Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaichen Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, Graham Neubig

MMedAgent: Learning to Use Medical Tools with Multi-modal Agent (2024.findings-emnlp)

Challenge:	Using leaderboards, researchers can track the performance of various systems on various NLP tasks.
Approach:	They propose a new conceptualization and implementation of NLP evaluation using a leaderboard.
Outcome:	The ExplainaBoard is an evaluation tool for natural language processing (NLP) it covers more than 400 systems, 50 datasets, 40 languages, and 12 tasks.

Copied to clipboard

Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, Yixin Wang

A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization (2023.acl-long)

Challenge:	Multi-modal Large Language Models (MLLMs) exhibit limited generality and often fall short when compared to specialized models.
Approach:	They propose a multi-modal medical agent that picks the most suitable medical tools based on user inputs.
Outcome:	The proposed agent performs better than open-source models and the closed-source model, GPT-4o.

Copied to clipboard

Lining Zhang, Simon Mille, Yufang Hou, Daniel Deutsch, Elizabeth Clark, Yixin Liu, Saad Mahamood, Sebastian Gehrmann, Miruna Clinciu, Khyathi Raghavi Chandu, João Sedoc

Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate (2023.findings-emnlp)

Challenge:	Using crowdsourcing, it is difficult to obtain high-quality annotations for difficult tasks.
Approach:	They propose a recruitment pipeline to recruit high-quality Amazon Mechanical Turk workers . they filter out subpar workers before they carry out the evaluations .
Outcome:	The proposed method can filter out subpar workers before they carry out evaluations and obtain high-agreement annotations with similar constraints on resources.

Copied to clipboard

Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, Bing Qin

Multi-Programming Language Sandbox for LLMs (2025.acl-demo)

Challenge:	Existing studies focus on inconsistency issues within a single LLM, while we explore the inter-consistencies among multiple LLMs for collaboration.
Approach:	They propose a formal debate framework to examine whether LLMs can collaborate effectively to achieve a consensus for a shared goal.
Outcome:	The proposed framework enables LLMs to achieve consensus in three real-world debate scenarios with real-time scenarios aligned to the LLM's goals.

Copied to clipboard

Shihan Dou, Jiazheng Zhang, Jianxiang Zang, Yunbo Tao, Weikang Zhou, Haoxiang Jia, Shichun Liu, Yuming Yang, Shenxi Wu, Zhiheng Xi, Muling Wu, Rui Zheng, Changze Lv, Limao Xiong, Shaoqing Zhang, Lin Zhang, Wenyu Zhan, Rongxiang Weng, Jingang Wang, Xunliang Cai, Yueming Wu, Ming Wen, Yixin Cao, Tao Gui, Xipeng Qiu, Qi Zhang, Xuanjing Huang

Leveraging Locality in Abstractive Text Summarization (2022.emnlp-main)

Challenge:	MPLSandbox is an out-of-the-box multi-programming language sandbox designed to provide unified and comprehensive feedback from compiler and analysis tools for Large Language Models (LLMs).
Approach:	They propose a multi-programming language sandbox that provides unified feedback from compilers and analysis tools for Large Language Models.
Outcome:	The proposed multi-language sandbox can provide comprehensive feedback from compilers and analysis tools for large language models (LLMs).

Copied to clipboard

Yixin Liu, Ansong Ni, Linyong Nan, Budhaditya Deb, Chenguang Zhu, Ahmed Hassan Awadallah, Dragomir Radev

Summarization

Challenge:	Neural attention models have improved on many natural language processing tasks, but their quadratic memory complexity hinders their applications in long text summarization.
Approach:	They propose to use a restricted context to study locality in text summarization . they propose to employ a quadratic memory growth with respect to the input length .
Outcome:	The proposed model has better performance than baseline models with efficient attention modules.

AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research (2025.acl-long)

Copied to clipboard

Yilun Zhao, Weiyuan Chen, Zhijian Xu, Manasi Patwardhan, Chengye Wang, Yixin Liu, Lovekesh Vig, Arman Cohan

LogicPro: Improving Complex Logical Reasoning via Program-Guided Learning (2025.acl-long)

Challenge:	a benchmark designed to evaluate the capabilities of LLMs in designing ablation studies for scientific research is available online.
Approach:	They propose to use a benchmark to evaluate LLMs' ability to design ablation studies . they investigate whether current automated evaluation methods are not reliable .
Outcome:	The benchmark compared leading LLMs with human experts on generating detailed ablation study designs . the results show that current evaluation methods are not reliable for the task .

Copied to clipboard

Jin Jiang, Yuchen Yan, Yang Liu, Jianing Wang, Shuai Peng, Xunliang Cai, Yixin Cao, Mengdi Zhang, Liangcai Gao

Planning Like Human: A Dual-process Framework for Dialogue Planning (2024.acl-long)

Challenge:	LogicPro is a data synthesis method that uses LeetCode-style algorithm problems and their corresponding Program solutions to generate complex logic data.
Approach:	They propose a new method which leverages LeetCode-style algorithm Problems and their corresponding Program solutions to synthesize complex logic data in text format.
Outcome:	The proposed method outperforms existing models for BBH27, LogicBench, DROP, AR-LSAT, and GSM8K, and a wide range of reasoning datasets.

Copied to clipboard

Tao He, Lizi Liao, Yixin Cao, Yuanxing Liu, Ming Liu, Zerui Chen, Bing Qin

Dialogue Systems & Conversational Agents

Challenge:	Large Language Models (LLMs) operate in a reactive mode, often resulting in efficiency issues or suboptimal performance.
Approach:	They propose a dual-process dialogue planning framework that leverages the dual-process theory of human cognition and a deliberative Monte Carlo Tree Search mechanism to emulate human-like conversational dynamics.
Outcome:	The proposed framework outperforms existing methods in achieving high-quality dialogues and operational efficiency.

Intuitive or Dependent? Investigating LLMs’ Behavior Style to Conflicting Prompts (2024.acl-long)

Copied to clipboard

Jiahao Ying, Yixin Cao, Kai Xiong, Long Cui, Yidong He, Yongbin Liu

RefSum: Refactoring Neural Summarization (2021.naacl-main)

Challenge:	Extensive experiments with seven Large Language Models reveal their varying behaviors.
Approach:	They investigate the behaviors of Large Language Models when faced with conflicting prompts versus their internal memory.
Outcome:	Extensive experiments with seven LLMs reveal their varying behaviors.

Copied to clipboard

Yixin Liu, Zi-Yi Dou, Pengfei Liu

Vector-Quantized Prompt Learning for Paraphrase Generation (2023.findings-emnlp)

Challenge:	Existing methods for text summarization are limited by limitations of reranking or stacking.
Approach:	They propose a framework that provides a unified view of text summarization and summaries combination.
Outcome:	The proposed method can be used by researchers as an off-the-shelf tool to achieve further performance improvements.

Copied to clipboard

Haotian Luo, Yixin Liu, Peidong Liu, Xianggen Liu

PersLEARN: Research Training through the Lens of Perspective Cultivation (2023.acl-demo)

Challenge:	Existing methods for paraphrase generation are difficult to understand and generate.
Approach:	They propose to generate diverse paraphrases by using instance-dependent prompts to control the generation of pre-trained models.
Outcome:	The proposed method achieves state-of-the-art on three benchmark datasets, including Quora, Wikianswers, and MSCOCO.

Copied to clipboard

Yu-Zhe Shi, Shiqian Li, Xinyi Niu, Qiao Xu, Jiawen Liu, Yifan Xu, Shiyu Gu, Bingru He, Xinyang Li, Xinyu Zhao, Zijian Zhao, Yidong Lyu, Zhen Li, Sijia Liu, Lin Qiu, Jinhao Ji, Lecheng Ruan, Yuxi Ma, Wenjuan Han, Yixin Zhu

Challenge:	PersLEARN is a tool designed to facilitate the cultivation of scientific perspectives . junior researchers struggle to identify the perspectives reflected in the literature and struggle to develop their own viewpoints.
Approach:	They propose a tool to facilitate the cultivation of scientific perspectives by interacting with a prompt-based model and allowing students to develop their own perspectives explicitly.
Outcome:	The proposed tool outperforms baseline approaches across multiple domains of literature from different perspectives.

Low-Resource Name Tagging Learned with Weakly Labeled Data (D19-1)

Copied to clipboard

Yixin Cao, Zikun Hu, Tat-seng Chua, Zhiyuan Liu, Heng Ji

Understanding Reference Policies in Direct Preference Optimization (2025.findings-naacl)

Challenge:	Existing methods for name tagging in low-resource languages or domains require extensive human efforts for training annotations.
Approach:	They propose a neural model for name tagging based on weakly labeled (WL) data.
Outcome:	The proposed model outperforms existing models in five low-resource languages and fine-grained food domains and shows that it is more efficient and efficient than existing models.

Copied to clipboard

Yixin Liu, Pengfei Liu, Arman Cohan

Expertise Style Transfer: A New Task Towards Better Communication between Experts and Laymen (2020.acl-main)

Challenge:	Direct Preference Optimization (DPO) is a widely used training method for the fine-tuning of large language models (LLMs).
Approach:	They explore the dependency on the reference model or policy in DPO and examine the necessity of the KL-constraint from the reference policies in D PO.
Outcome:	The proposed model is sensitive to the KL divergence constraint, and the reference policy can penalize deviations from the reference constraint.

Copied to clipboard

Yixin Cao, Ruihao Shui, Liangming Pan, Min-Yen Kan, Zhiyuan Liu, Tat-Seng Chua

R2D2: Robust Data-to-Text with Replacement Detection (2022.emnlp-main)

Challenge:	a curse of knowledge can impede communication between experts and laymen.
Approach:	They propose a task of expertise style transfer to automate the communication between experts and laymen using an annotated dataset.
Outcome:	The proposed task simplifies professional language and improves accuracy and expertise level of laymen descriptions using simple words.

Copied to clipboard

Linyong Nan, Lorenzo Jaime Flores, Yilun Zhao, Yixin Liu, Luke Benson, Weijin Zou, Dragomir Radev

Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation (2023.emnlp-main)

Challenge:	Existing methods to mitigate unfaithful text generation are inadequate . data-to-text generation requires a structured input format .
Approach:	They propose a training framework that addresses unfaithful Data-to-Text generation by training a system as a generator and faithfulness discriminator with additional replacement detection and unlikelihood learning tasks.
Outcome:	The proposed training framework improves FeTaQA, LogicNLG, and ToTTo fidelity on D2T systems.

Copied to clipboard

Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev

On Learning Text Style Transfer with Direct Rewards (2021.naacl-main)

Challenge:	Compared to neural systems, automatic metrics should be interpretable and provide intuitive insights into system performance and output quality.
Approach:	They propose to use a two-stage evaluation pipeline to extract basic information units from one text sequence and check the extracted units in another sequence.
Outcome:	The proposed metrics can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability.

Copied to clipboard

Yixin Liu, Graham Neubig, John Wieting

ReIFE: Re-evaluating Instruction-Following Evaluation (2025.naacl-long)

Challenge:	Existing methods for text style transfer lack parallel corpora, which makes it impossible to train supervised models.
Approach:	They propose to use semantic similarity metrics to explicitly assess the preservation of content between system outputs and inputs.
Outcome:	The proposed methods provide significant gains in automatic and human evaluation over strong baselines.

Copied to clipboard

Yixin Liu, Kejian Shi, Alexander Fabbri, Yilun Zhao, PeiFeng Wang, Chien-Sheng Wu, Shafiq Joty, Arman Cohan

Robustness in NLP

Challenge:	Existing evaluations of large language models (LLMs) for instruction following are incomplete.
Approach:	They propose to use 25 base LLMs and 15 recently proposed evaluation protocols to evaluate instruction following on 4 human-annotated datasets.
Outcome:	The proposed evaluations identify the best-performing base LLMs and evaluation protocols with a high degree of robustness.

Fair Abstractive Summarization of Diverse Perspectives (2024.naacl-long)

Copied to clipboard

Yusen Zhang, Nan Zhang, Yixin Liu, Alexander Fabbri, Junru Liu, Ryo Kamoi, Xiaoxin Lu, Caiming Xiong, Jieyu Zhao, Dragomir Radev, Kathleen McKeown, Rui Zhang

Rethinking Efficient Multilingual Text Summarization Meta-Evaluation (2024.findings-acl)

Challenge:	Existing work on summarization metrics and large language models has not explored fair abstractive summarizing.
Approach:	They propose four reference-free automatic metrics to measure the differences between target and source perspectives.
Outcome:	The proposed methods alleviate fair abstractive summarization on user-generated data.

Copied to clipboard

Rilyn Han, Jiawen Chen, Yixin Liu, Arman Cohan

CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalization (2026.acl-long)

Challenge:	a limited number of human annotations are required to evaluate multilingual summarization evaluation metrics.
Approach:	They propose a multilingual meta-evaluation framework that uses machine translation systems to transform a monolingual metaevaluations dataset into multilingual versions.
Outcome:	The proposed framework outperforms classical text-matching-based metrics in non-English languages.

Copied to clipboard

Zhongyuan Peng, Yifan Yao, Kaijing Ma, Shuyue Guo, Yizhe Li, Yichi Zhang, Chenchen Zhang, Yifan Zhang, Zhouliang Yu, Luming Li, Minghao Liu, Yihang Xia, Jiawei Shen, Yuchen Wu, Yixin Cao, Zhaoxiang Zhang, Wenhao Huang, Jiaheng Liu, Ge Zhang

Multi-Channel Graph Neural Network for Entity Alignment (P19-1)

Challenge:	Existing approaches to formalizing mathematical statements face limitations in accuracy, especially in the context of complex, highlevel problems that involve sophisticated mathematical reasoning.
Approach:	They propose a CriticLean framework that elevates the role of the critic from a passive validator to an active learning component and introduce a benchmark to measure models’ ability to distinguish semantically correct from incorrect formalizations.
Outcome:	The proposed framework outperforms open- and closed-source benchmarks and shows that it significantly outperformed existing models.

Copied to clipboard

Yixin Cao, Zhiyuan Liu, Chengjiang Li, Zhiyuan Liu, Juanzi Li, Tat-Seng Chua

Structured Data in NLP

Challenge:	Existing methods to learn alignment-oriented knowledge graph embeddings suffer from structural heterogeneity and limited seed alignments.
Approach:	They propose a multi-channel Graph Neural Network model to learn alignment-oriented knowledge graph embeddings by encoding two KGs via multiple channels.
Outcome:	The proposed model is expected to reconcile the structural differences of two KGs, and thus make better use of seed alignments.

Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference (2025.findings-naacl)

Copied to clipboard

Mingqi Gao, Yixin Liu, Xinyu Hu, Xiaojun Wan, Jonathan Bragg, Arman Cohan

Challenge:	Evaluating and ranking the capabilities of different LLMs is crucial for understanding their performance and alignment with human preferences.
Approach:	They propose a system-level evaluation framework that ranks LLMs based on their alignment with human preferences.
Outcome:	The proposed framework aims to rank LLMs based on their performance and alignment with human preferences.

On the Role of Summary Content Units in Text Summarization Evaluation (2024.naacl-short)

Copied to clipboard

Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo Walenta, Juri Opitz, Leonardo Ribeiro, João Sedoc, Daniel Deutsch, Simon Mille, Yixin Liu, Sebastian Gehrmann, Lining Zhang, Saad Mahamood, Miruna Clinciu, Khyathi Chandu, Yufang Hou

Summarization

Challenge:	a human written summary content unit (SCU) is used to judge the quality of a summary . a pyramid evaluation method is based on SCUs that decompose a reference summary into concise sentences .
Approach:	They propose to use automated SCUs to evaluate the quality of a candidate summary . they propose to generate SCU approximations from AMR meaning representations and large language models .
Outcome:	The proposed method can be fully automated, but lacks the human effort to validate it.

Exploring and Evaluating Attributes, Values, and Structures for Entity Alignment (2020.emnlp-main)

Copied to clipboard

Zhiyuan Liu, Yixin Cao, Liangming Pan, Juanzi Li, Zhiyuan Liu, Tat-Seng Chua

Structured Data in NLP

Challenge:	Entity alignment (EA) aims at building a Knowledge Graph (KG) of rich content by linking the equivalent entities from various KGs.
Approach:	They propose to use an attributed value encoder to partition a Knowledge Graph into subgraphs to model the various types of attribute triples efficiently.
Outcome:	The proposed method achieves significant improvements over 12 baselines in cross-lingual and monolingual datasets.

On Improving Summarization Factual Consistency from Natural Language Feedback (2023.acl-long)

Copied to clipboard

Yixin Liu, Budhaditya Deb, Milagro Teruel, Aaron Halfaker, Dragomir Radev, Ahmed Hassan Awadallah

GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022.emnlp-demos)

Challenge:	Recent work shows that language generation models can make errors on fine-grained qualities such as factual consistency.
Approach:	They propose to use natural language feedback to improve generation quality and user preference alignment.
Outcome:	The proposed model can provide factual consistency in human-edited summaries and further insights into summarization factual consistentness.

Copied to clipboard

Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F . R. Ribeiro, Lewis Tunstall, Li Zhang, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou

Breaking the Reasoning Barrier A Survey on LLM Complex Reasoning through the Lens of Self-Evolution (2025.findings-acl)

Challenge:	Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work.
Approach:	They propose to use the Generation, Evaluation, and Metrics Benchmark to integrate new evaluation methods into existing evaluations.
Outcome:	The proposed evaluation infrastructure bridges the gap between the advantages of leaderboards and in-depth and evolving evaluations by allowing model developers to benefit from each other's work.

Copied to clipboard

Tao He, Hao Li, Jingchang Chen, Runxuan Liu, Yixin Cao, Lizi Liao, Zihao Zheng, Zheng Chu, Jiafeng Liang, Ming Liu, Bing Qin

Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving (2025.findings-acl)

Challenge:	OpenAI's O1 and subsequent projects like DeepSeek R1 have significantly advanced research on complex reasoning in LLMs.
Approach:	They analyze existing reasoning studies from the perspective of self-evolution and summarize O1-like works from open-source projects like DeepSeek R1 and Kimi-k1.5.
Outcome:	The proposed models are based on open-source models and pioneer advanced methodologies like Scaling Reinforcement Learning (RL).

Copied to clipboard

Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, Arman Cohan

Challenge:	a benchmark for university-level physics problem solving contains 1,297 expert-annotated problems . a proprietary model, o3-mini, achieves only 59.9% accuracy, highlighting fundamental weaknesses in scientific reasoning, conceptual understanding, and mathematical precision.
Approach:	They introduce Physics, a benchmark for university-level physics problem solving.
Outcome:	The proposed model achieves only 59.9% accuracy on the most advanced model, o3-mini . the proposed model is a powerful tool for evaluating models on advanced problems .

Neural Collective Entity Linking (C18-1)

Copied to clipboard

Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu

Knowledge Representation

Challenge:	Entity linking aims to link entity mentions in texts to knowledge bases, but existing methods rely on local contexts to resolve entities independently.
Approach:	They propose a neural model for collective entity linking that integrates local contextual features and global coherence information to improve the computation efficiency.
Outcome:	The proposed model improves its performance on five publicly available datasets and can be used to train on Wikipedia hyperlinks to avoid overfitting and domain bias.

X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects (2024.naacl-long)

Copied to clipboard

Minqian Liu, Ying Shen, Zhiyang Xu, Yixin Cao, Eunah Cho, Vaibhav Kumar, Reza Ghanadan, Lifu Huang

Calibrating Long-form Generations From Large Language Models (2024.findings-emnlp)

Challenge:	X-Eval is a two-stage instruction tuning framework to evaluate text in both seen and unseen aspects customized by end users.
Approach:	They introduce a two-stage instruction tuning framework to evaluate text in both seen and unseen aspects customized by end users.
Outcome:	The proposed framework improves the model’s ability to follow evaluation instructions and enhances the learning stage to better assess text quality.

Copied to clipboard

Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, Bhuwan Dhingra

Is Multi-Hop Reasoning Really Explainable? Towards Benchmarking Reasoning Interpretability (2021.emnlp-main)

Challenge:	Conventional calibration methods treat answer correctness as binary and do not work for long-form generation where an answer can be partially correct.
Approach:	They propose a framework where correctness of LLMs' responses and associated confidence levels are treated as distributions across a range of scores.
Outcome:	The proposed framework treats the correctness of the LLMs’ responses and their associated confidence levels as distributions across a range of scores.

Copied to clipboard

Xin Lv, Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Yichi Zhang, Zelin Dai

CHEER: Centrality-aware High-order Event Reasoning Network for Document-level Event Causality Identification (2023.acl-long)

Challenge:	Existing models for multi-hop reasoning are not able to evaluate their interpretability . a recent study found that many paths are unreasonable .
Approach:	They propose a framework to evaluate the interpretability of multi-hop reasoning models . they annotate all possible rules and establish a benchmark .
Outcome:	The proposed framework outperforms existing models in terms of performance and interpretability.

Copied to clipboard

Meiqi Chen, Yixin Cao, Yan Zhang, Zhiwei Liu

SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization (2021.acl-short)

Challenge:	Recent studies focus on building a document-level graph for cross-sentence reasoning, but ignore important causal structures.
Approach:	They propose a document-level event causality identification model which annotates central events and incorporates event centrality information into the reasoning network.
Outcome:	The proposed model performs high-order reasoning while considering event centrality.

Copied to clipboard

Yixin Liu, Pengfei Liu

Explainable and Fine-Grained Safeguarding of LLM Multi-Agent Systems via Bi-Level Graph Anomaly Detection (2026.acl-long)

Challenge:	Experimental results show that SimCLS can improve existing top-performing models by a large margin.
Approach:	They propose a framework for abstractive summarization that is conceptually simple and empirically powerful.
Outcome:	The proposed framework improves the performance of top-performing models by a large margin against existing top-scoring systems.

Copied to clipboard

Junjun Pan, Yixin Liu, Rui Miao, Kaize Ding, Yu Zheng, Quoc Viet Hung Nguyen, Alan Wee-Chung Liew, Shirui Pan

Do Pre-trained Models Benefit Knowledge Graph Completion? A Reliable Evaluation and a Reasonable Approach (2022.findings-acl)

Challenge:	Existing graph anomaly detection methods rely on coarse sentence-level information and overlook fine-grained lexical cues, limiting their reliability and real-world applicability.
Approach:	They propose an explainable and fine-grained safeguarding framework for detecting malicious agents in multi-agent systems (MAS) to incorporate both coarse and fine lexical information for anomalous agent identification.
Outcome:	Extensive experiments across diverse MAS topologies and attack scenarios demonstrate robust detection performance and strong interpretability of XG-Guard.

Copied to clipboard

Xin Lv, Yankai Lin, Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, Jie Zhou

BRIO: Bringing Order to Abstractive Summarization (2022.acl-long)

Challenge:	Pre-trained language models capture factual knowledge from massive texts . but they are still quite behind the SOTA KGC models in terms of performance .
Approach:	They propose to use open-world assumption to evaluate PLM-based knowledge graph completion models . they propose to convert each triple and its support information into natural prompt sentences .
Outcome:	The proposed model is more accurate under the open-world assumption (OWA) this setting manual checks the correctness of knowledge that is not in KGs.

Copied to clipboard

Yixin Liu, Pengfei Liu, Dragomir Radev, Graham Neubig

Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization (2024.findings-naacl)

Challenge:	Abstractive summarization models are often trained with maximum likelihood estimation (MLE) . mLE assumes a deterministic (one-point) target distribution, but can cause performance degradation .
Approach:	They propose a new training paradigm which assumes a non-deterministic distribution so that different candidate summaries are assigned probability mass according to their quality.
Outcome:	The proposed model can estimate probabilities of candidate summaries that are more correlated with their level of quality.

Copied to clipboard

Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, Arman Cohan

Information Extraction & Text Mining

Challenge:	Recent studies have found that large language models (LLMs) can achieve state-of-the-art performance on generic summarization benchmarks, but their performance on more complex summarizing task settings is less studied.
Approach:	They benchmark large language models on instruction controllable text summarization . they use 4 evaluation protocols and 11 LLMs to evaluate their performance .
Outcome:	The proposed model performs well on instruction controllable text summarization tasks with 4 evaluation protocols and 11 LLMs.

CoVariance-based Causal Debiasing for Entity and Relation Extraction (2023.findings-emnlp)

Copied to clipboard

Lin Ren, Yongbin Liu, Yixin Cao, Chunping Ouyang

Information Extraction & Text Mining

Challenge:	Named Entity Recognition and Relation Extraction are key tasks of Information Extraction.
Approach:	They propose a causal framework called c ovariance and variance optimization framework (OVO) to optimize feature representations and conduct general debiasing.
Outcome:	The proposed framework minimizes characterizing features’ covariance for alleviating selection and distribution bias and enhances feature representation in the feature space.

Uncertainty Calibration for Tool-Using Language Agents (2024.findings-emnlp)

Copied to clipboard

Hao Liu, Zi-Yi Dou, Yixin Wang, Nanyun Peng, Yisong Yue

CourtReasoner: Can LLM Agents Reason Like Judges? (2025.emnlp-main)

Challenge:	Language agents are increasingly used to perform tasks and interact with a variety of external tools to achieve specific, goal-oriented objectives.
Approach:	They propose a tool calibration tool called ProbeCal which recalibrates the internal probabilities of tool-using language agents to better reflect the actual effectiveness of tool.
Outcome:	The proposed model significantly improves off-the-shelf language models in tool-using applications.

Copied to clipboard

Sophia Simeng Han, Yoshiki Takashima, Shannon Zejiang Shen, Chen Liu, Yixin Liu, Roque K. Thuo, Sonia Knowlton, Ruzica Piskac, Scott J Shapiro, Arman Cohan

M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models (2024.findings-emnlp)

Challenge:	Large language models (LLMs) have achieved impressive results across mathematical, logical, and commonsense reasoning tasks.
Approach:	They propose a novel expert-annotated judicial reasoning benchmark to measure LLMs' ability to construct goal-oriented legal reasoning.
Outcome:	The proposed benchmark measures the LLM agent’s ability to construct goal-oriented legal reasoning.

Copied to clipboard

Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, Arman Cohan

Text Clustering

Challenge:	Existing evaluation benchmarks for foundation models in understanding scientific literature focus on single-document tasks.
Approach:	They propose a multi-modal, multi-document scientific question answering benchmark . it uses expert-annotated questions that span 70 natural language processing paper clusters .
Outcome:	The proposed benchmarks underperform human experts in multi-modal reasoning and retrieval of scientific data.

IPL: Leveraging Multimodal Large Language Models for Intelligent Product Listing (2024.emnlp-industry)

Copied to clipboard

Kang Chen, Qing Zhang, Chengbao Lian, Yixin Ji, Xuwei Liu, Shuguang Han, Guoqiang Wu, Fei Huang, Jufeng Chen

Com2 : A Causal-Guided Benchmark for Exploring Complex Commonsense Reasoning in Large Language Models (2025.acl-long)

Challenge:	Unlike professional Business-to-Consumer (B2C) e-commerce platforms, consumer-to consumer (C2C), is mainly targeting individual sellers.
Approach:	They develop an intelligent product listing tool that generates product descriptions using various product attributes such as category, brand, color, condition, etc.
Outcome:	The proposed tool outperforms the base model in domain-specific tasks while producing less hallucination.

Copied to clipboard

Kai Xiong, Xiao Ding, Yixin Cao, Yuxiong Yan, Li Du, Yufei Zhang, Jinglong Gao, Jiaqian Liu, Bing Qin, Ting Liu

Learning from Miscellaneous Other-Class Words for Few-shot Named Entity Recognition (2021.acl-long)

Challenge:	Existing works focus on complex tasks like math and code, while complex commonsense reasoning remains underexplored due to its uncertainty and lack of structure.
Approach:	They propose to build a benchmark for large language models based on complex commonsense reasoning based upon causal event graphs and causal theory.
Outcome:	The proposed benchmark combines a complex commonsense reasoning benchmark with a detective story to achieve a more challenging subset.

Copied to clipboard

Meihan Tong, Shuai Wang, Bin Xu, Yixin Cao, Minghui Liu, Lei Hou, Juanzi Li

Low-Resource NLP

Challenge:	Existing methods to classify named entity mentions with fewshots fail to differentiate rich semantics in other-class words, which will aggravate overfitting under few shot scenario.
Approach:	They propose a model that can automatically induce different unde- fined classes from the other class to improve few-shot Named Entity Recognition (NER) .
Outcome:	The proposed model outperforms five state-of-the-art models in 1- shot and 5-shots settings on four NER bench marks.

DUD: Decoupled Update Dynamics for Reliable Uncertainty Quantification in Large Language Models (2026.acl-long)

Copied to clipboard

Yixin Bu, Runze Xia, Guanyun Zou, Yupeng Ji, Hongliang Dai, Haodong Liu, Piji Li

UNICORN: A Unified Causal Video-Oriented Language-Modeling Framework for Temporal Video-Language Tasks (2024.emnlp-main)

Challenge:	Accurate Uncertainty Quantification (UQ) is critical for reliable deployment of Large Language Models (LLMs).
Approach:	They propose a framework that explicitly decouples FFN and Attention contributions via noise-induced causal interventions to capture model's internal fragility.
Outcome:	The proposed framework outperforms state-of-the-art baselines in both uncertainty estimation and calibration while exhibiting superior cross-dataset generalization.

Copied to clipboard

Yuanhao Xiong, Yixin Nie, Haotian Liu, Boxin Wang, Jun Chen, Rong Jin, Cho-Jui Hsieh, Lorenzo Torresani, Jie Lei

DataLab: A Platform for Data Analysis and Intervention (2022.acl-demo)

Challenge:	Recent advances in large multimodal models have encouraged the development of large multi-modal models . however, it is unclear how to extend these models to the more complex video domain .
Approach:	They propose a visual instruction tuning framework to address temporal video-language tasks . they collect a dataset and fine-tune the framework on instruction-following data .
Outcome:	The proposed model can perform better on established temporal video-language tasks without training objectives and intensive pre-training.

Copied to clipboard

Yang Xiao, Jinlan Fu, Weizhe Yuan, Vijay Viswanathan, Zhoumianze Liu, Yixin Liu, Graham Neubig, Pengfei Liu

Text-guided 3D Human Generation from 2D Collections (2023.findings-emnlp)

Challenge:	Existing tools and research focus on how to interpret and manipulate data, despite its crucial role in machine learning, . existing tools and researchers focus on systems on top of existing data, rather than how to use it.
Approach:	They propose a unified data-oriented platform that allows users to interactively analyze the characteristics of data and provides a standard interface for many data processing operations.
Outcome:	The proposed platform allows users to analyze the characteristics of data and provides a standardized interface so that many data processing operations can be provided within a single interface.

Copied to clipboard

Tsu-Jui Fu, Wenhan Xiong, Yixin Nie, Jingyu Liu, Barlas Oguz, William Wang

Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2023.acl-long)

Challenge:	3D human modeling is used for engaging interaction in gaming, film, and animation. however, the customization of characters is crucial for creativity and scalability.
Approach:	They propose a 3D human generation using fashion descriptions to enhance 3D geometry transformation and fine-grained consistency.
Outcome:	The proposed model can generate a 3D human, guided by a fashion description, with high efficiency.

Copied to clipboard

Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev

Robustness in NLP

Challenge:	Existing studies for summarization evaluation exhibit low inter-annotator agreement or lack scale.
Approach:	They propose a modified summarization salience protocol based on fine-grained semantic units and a robust summarizing evaluation benchmark.
Outcome:	The proposed protocol is based on fine-grained semantic units and allows for high inter-annotator agreement.

QTSumm: Query-Focused Summarization over Tabular Data (2023.emnlp-main)

Copied to clipboard

Yilun Zhao, Zhenting Qi, Linyong Nan, Boyu Mi, Yixin Liu, Weijin Zou, Simeng Han, Ruizhe Chen, Xiangru Tang, Yumo Xu, Dragomir Radev, Arman Cohan

Knowledge Control for Responsible Generative AI: Bridging Academia, Industry, and Society (2026.acl-tutorials)

Challenge:	Existing text generation systems that can provide accurate table summaries can facilitate more efficient access to relevant data insights.
Approach:	They propose a query-focused task where text generation models have to perform human-like reasoning and analysis over the given table to generate a tailored table summary.
Outcome:	The proposed method improves existing baselines on table-to-text generation and large language models by concatenating generated facts to the model input.

Copied to clipboard

Zheyuan Liu, Yixin Wan, Kai-Wei Chang, Meng Jiang, Jieyu Zhao, Nouha Dziri, Yuning Mao, Jia-Chen Gu, Jindong Gu

CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search (2025.naacl-industry)

Challenge:	This tutorial introduces the foundations of post-training knowledge control and showcases recent frontier methods.
Approach:	This tutorial introduces the foundations of post-training knowledge control and showcases recent frontier methods.
Outcome:	This tutorial introduces the foundations of post-training knowledge control and showcases recent frontier methods . key motivations and failure modes, harmful generation and stereotype reinforcement, are addressed . core methods such as machine unlearning, knowledge editing, and inference-time interventions are also included .

Copied to clipboard

Kaixin Wu, Yixin Ji, Zeyuan Chen, Qiang Wang, Cunxiang Wang, Hong Liu, Baijun Ji, Xu Jia, Zhongyi Liu, Jinjie Gu, Yuan Zhou, Linjian Mo

Are LLMs Rational Investors? A Study on the Financial Bias in LLMs (2025.findings-acl)

Challenge:	Relevance modeling between queries and items is a key component of commercial search engines.
Approach:	They propose a framework for continual pre-training of LLMs to enhance domain knowledge . they employ queries and multi-field item to jointly pre-train for enhancing domain knowledge.
Outcome:	The proposed model achieves convincing performance compared to strong baselines.

Copied to clipboard

Yuhang Zhou, Yuchen Ni, Zhiheng Xi, Zhangyue Yin, Yu He, Gan Yunhui, Xiang Liu, Zhang Jian, Sen Liu, Xipeng Qiu, Yixin Cao, Guangnan Ye, Hongfeng Chai