Papers by Ye Qi

Muse: Towards Reproducible Long-Form Song Generation with Fine-Grained Style Control (2026.findings-acl)

Challenge:	Existing methods for code retrieval struggle to balance scalability and annotation quality.
Approach:	They propose a method that integrates functions called within the repository and information on third-party APIs to enhance the annotation context.
Outcome:	The proposed method improves the annotation context by incorporating functions called within the repository and information on third-party API functionalities.

Copied to clipboard

Changhao Jiang, Jiahao Chen, Zhenghao Xiang, Zhixiong Yang, Hanchen Wang, Jiabao Zhuang, Xinmeng Che, Jiajun Sun, Hui Li, Yifei Cao, Shihan Dou, Ming Zhang, Junjie Ye, Tao Ji, Tao Gui, Qi Zhang, Xuanjing Huang

Speech & Audio in NLP

Challenge:	Recent commercial systems such as Suno demonstrate strong capabilities in long-form song generation, but academic research remains non-reproducible due to the lack of publicly available training data.
Approach:	They propose a system for long-form song generation with fine-grained style conditioning that includes a licensed synthetic dataset and a song generation model, Muse.
Outcome:	The proposed system achieves competitive performance on phoneme error rate, text–music style similarity, and audio aesthetic quality while enabling controllable segment-level generation across different musical structures.

Augmenting Multi-Turn Text-to-SQL Datasets with Self-Play (2022.findings-emnlp)

Copied to clipboard

Qi Liu, Zihuiwen Ye, Tao Yu, Linfeng Song, Phil Blunsom

Programming Languages in NLP

Challenge:	Numerous architectures and pretraining methods have been proposed for context-dependent text-to-SQL, but the size of the datasets used has been limited due to the high cost of annotating multi-turn dialogue and SQL pairs.
Approach:	They propose to augment training datasets using self-play which leverages contextual information to synthesize new interactions to adapt the model to new databases.
Outcome:	The proposed model improves accuracy on SParC and CoSQL, two widely used cross-domain text-to-SQl datasets.

FinToolSyn: A forward synthesis Framework for Financial Tool-Use Dialogue Data with Dynamic Tool Retrieval (2026.findings-acl)

Copied to clipboard

Caishuang Huang, Yang Qiao, Rongyu Zhang, Junjie Ye, Pu Lu, null Wuwenxi, Meng Zhou, Xiku Du, Qi Zhang, Tao Gui, Xuanjing Huang

Keyphrase Generation with Fine-Grained Evaluation-Guided Reinforcement Learning (2021.findings-emnlp)

Challenge:	Existing data synthesis methods rely on static tools to generate queries . this approach fails to capture the implicit, event-driven nature of real-world needs .
Approach:	They propose a forward synthesis framework to generate high-quality financial dialogues . they construct a repository of 43,066 tools and synthesize over 148k dialogue instances .
Outcome:	Experiments show that models trained on FinToolSyn achieve a 21.06% improvement . the framework is designed to generate high-quality financial dialogues .

Copied to clipboard

Yichao Luo, Yige Xu, Jiacheng Ye, Xipeng Qiu, Qi Zhang

Structured Data in NLP

Challenge:	Existing KG evaluation metrics are only aware of the exact correctness of predictions on phrase-level and ignore semantic similarities between similar predictions and targets, which inhibits the model from learning deep linguistic patterns.
Approach:	They propose a fine-grained evaluation metric to improve the previous KG framework . the evaluation metrics are only aware of the exact correctness of predictions on phrase-level .
Outcome:	The proposed method outperforms the existing frameworks among all evaluation scores.

CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extending (2024.acl-long)

Copied to clipboard

Shiyi Zhu, Jing Ye, Wei Jiang, Siqiao Xue, Qi Zhang, Yifan Wu, Jianguo Li

CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios (2025.emnlp-main)

Challenge:	Existing models that use self-attention and position embedding have anomalous behavior that hinder long context window extrapolation.
Approach:	They propose a collinear constraint between Q and K to integrate RoPE and self-attention.
Outcome:	The proposed model integrates self-attention and position embedding into LLMs without fine-tuning.

Copied to clipboard

Shiting Huang, Zhen Fang, Zehui Chen, Siyu Yuan, Junjie Ye, Yu Zeng, Lin Chen, Qi Mao, Feng Zhao

FOLIO: Natural Language Reasoning with First-Order Logic (2024.emnlp-main)

Challenge:	a number of tools are used to perform complex tasks, but the tool utilization process can cause errors.
Approach:	They propose a critique evaluation benchmark for tool learning that analyzes function-calling errors on tool evaluation benchmarks.
Outcome:	The proposed critique evaluation benchmark holds diverse tool-use errors with varying complexities, which better reflects real-world scenarios.

Copied to clipboard

Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, Dragomir Radev

Reasoning

Challenge:	Existing benchmarks for logical reasoning in large language models lack language naturalness or limited complexity.
Approach:	They propose to use first-order logic annotations to evaluate logical reasoning capabilities of large language models.
Outcome:	The proposed dataset evaluates the FOL reasoning ability of supervised fine-tuning on medium-sized language models.

Llama SLayer 8B: Shallow Layers Hold the Key to Knowledge Injection (2024.findings-emnlp)

Copied to clipboard

Tianxiang Chen, Zhentao Tan, Tao Gong, Yue Wu, Qi Chu, Bin Liu, Jieping Ye, Nenghai Yu

Challenge:	Existing methods to augment pre-trained large language models require extensive computational efforts and massive data volumes, challenging the widespread accessibility of LLM research.
Approach:	They propose a post-pretraining strategy of selectively enhancing shallow layers while pruning less effective deep ones to augment pretrained large language models.
Outcome:	The proposed approach improves performance on the corpus of code & math and a legal corpus and is widely applicable.

AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments (2026.acl-long)

Copied to clipboard

Zhiheng Xi, Dingwen Yang, Jiaqi Liu, Jixuan Huang, Honglin Guo, Baodai Huang, Tinggang Chen, Qi Zhang, Zhonghang Lu, Chenyu Liu, Jiajun Sun, Jiazheng Zhang, Dingwei Zhu, Xin Guo, Junzhe Wang, Zhihao Zhang, Yuming Yang, Junjie Ye, Minghe Gao, Dongrui Liu, Jiaming Ji, Guohao Li, Tao Gui, Qi Zhang, Xuanjing Huang

Argue with Me Tersely: Towards Sentence-Level Counter-Argument Generation (2023.emnlp-main)

Challenge:	Existing benchmarks evaluate agents in simplified, idealized settings, relying on pre-packaged tool interfaces, overlooking critical steps, and assume inputs are clean and fully specified.
Approach:	They propose a framework that evaluates language agents in simplified, idealized settings . they show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 .
Outcome:	Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 .

Copied to clipboard

Jiayu Lin, Rong Ye, Meng Han, Qi Zhang, Ruofei Lai, Xinyu Zhang, Zhao Cao, Xuanjing Huang, Zhongyu Wei

Text Generation

Challenge:	Existing work describes paragraph-level counter-argument generation task as paragraph-based . however, sentence-level generation can be quite different due to its unique constraints and brevity-focused challenges.
Approach:	They propose a benchmark framework for sentence-level counter-argument generation . they use an annotated debate forum dataset to generate high-quality counter-argments .
Outcome:	The proposed framework and evaluator are competitive in counter-argument generation tasks.

Causal Intervention Improves Implicit Sentiment Analysis (2022.coling-1)

Copied to clipboard

Siyin Wang, Jie Zhou, Changzhi Sun, Junjie Ye, Tao Gui, Qi Zhang, Xuanjing Huang

Sentiment Analysis

Challenge:	Existing neural models struggle with implicit sentiment analysis because they latch onto spurious correlations, resulting in poor generalization and robustness.
Approach:	They propose a CausaL intervention model for implicit sEntiment ANalysis using instrumental variable to eliminate confounding causal effects and extract the pure causal effect between sentence and sentiment.
Outcome:	The proposed model extracts the pure causal effect between sentence and sentiment using instrumental variable.

Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization (2024.findings-acl)

Copied to clipboard

Yanghai Zhang, Ye Liu, Shiwei Wu, Kai Zhang, Xukai Liu, Qi Liu, Enhong Chen

One Battle After Another: Probing LLMs’ Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework (2026.acl-long)

Challenge:	Multimodal Summarization with Multimodal Output (MSMO) is a new approach to produce a multimodal summary that integrates both text and relevant images.
Approach:	They propose an Entity-Guided Multimodal Summarization model that integrates both text and relevant images to produce a multimodal summary.
Outcome:	The proposed model integrates text-image and entity-image information and refines image selection through knowledge distillation from a pre-trained vision-language model.

Copied to clipboard

Qi Jia, Ye Shen, Xiujie Song, Kaiwei Zhang, Shibo Wang, Dun Pei, Xiangyang Zhu, Guangtao Zhai

GeAR: Graph-enhanced Agent for Retrieval-augmented Generation (2025.findings-acl)

Challenge:	Existing benchmarks for instruction-following in multi-topic dialogues are limited to a fixed number of turns, susceptible to saturation and failing to account for users’ interactive experience.
Approach:	They propose a framework featuring a three-layer tracking mechanism and a query synthesis agent to mimic sequential user behaviors.
Outcome:	The proposed framework outperforms existing benchmarks in the evaluation of instruction following in multi-topic dialogues and demonstrates deficiencies in failure recovery and fine-grained instruction following.

Copied to clipboard

Zhili Shen, Chenxin Diao, Pavlos Vougiouklis, Pascual Merita, Shriram Piramanayagam, Enting Chen, Damien Graux, Andre Melo, Ruofei Lai, Zeren Jiang, Zhongyang Li, Ye Qi, Yang Ren, Dandan Tu, Jeff Z. Pan

Improving Discriminative Capability of Reward Models in RLHF Using Contrastive Learning (2024.emnlp-main)

Challenge:	Retrieval-augmented Generation (RAG) relies on effective retrieval capabilities, yet traditional sparse and dense retrievers struggle with multi-hop retrieval scenarios.
Approach:	They propose a graph expansion mechanism that augments any conventional base retriever and an agent framework that incorporates the resulting graph-based retrieval into a multi-step retrieval framework.
Outcome:	The proposed system achieves state-of-the-art results on three multi-hop question answering datasets while consuming fewer tokens and requiring fewer iterations than existing multi-step retrieval systems.

Copied to clipboard

Lu Chen, Rui Zheng, Binghai Wang, Senjie Jin, Caishuang Huang, Junjie Ye, Zhihao Zhang, Yuhao Zhou, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang

DARM: Distribution-Aware Reward Modeling by Alleviating Biases from Low Preference-Context Dependency Data (2026.acl-long)

Challenge:	Current methods rely on ranking losses to teach reward model to assess preferences, but they are susceptible to noise and ambiguous data, often failing to deeply understand human intentions.
Approach:	They propose a method that incorporates contrastive learning into the reward modeling process to enhance generalization and stabilize the reinforcement learning training process.
Outcome:	The proposed method enhances generalization of the reward model, stabilizes the reinforcement learning training process, and improves the final alignment with human preferences.

Copied to clipboard

Shaofan Liu, Guoqiang Zhang, Shihan Dou, Huiyuan Zheng, Yiming Zhou, Junjie Ye, Shaowen Wang, Shichun Liu, Jiazheng Zhang, Tao Gui, Qi Zhang, Xuanjing Huang

Challenge:	Existing methods for training reward models are vulnerable to context neglect and degraded accuracy.
Approach:	They propose distribution-aware reward modeling that augments the RM objective with a conditional mutual information regularizer that maximizes context and the predicted reward conditioned on the response.
Outcome:	The proposed model improves performance in RLHF and improves accuracy in other settings.

TransferTOD: A Generalizable Chinese Multi-Domain Task-Oriented Dialogue System with Transfer Capabilities (2024.emnlp-main)

Copied to clipboard

Ming Zhang, Caishuang Huang, Yilong Wu, Shichun Liu, Huiyuan Zheng, Yurui Dong, Yujiong Shen, Shihan Dou, Jun Zhao, Junjie Ye, Qi Zhang, Tao Gui, Xuanjing Huang

Travel on the ICD Tree: Benchmarking Agentic Reasoning for ICD Coding from Chinese Electronic Medical Records (2026.findings-acl)

Challenge:	Current datasets cater to user-led systems and are limited to predefined specific scenarios and slots.
Approach:	They propose to use a Chinese dialogue dataset to train a model that authentically simulates human-computer dialogues in 30 popular life service scenarios.
Outcome:	The proposed model achieves a joint accuracy of 75.09% in out-of-domain evaluations . it also achieves notable abilities in slot filling and questioning .

Copied to clipboard

Xinjie Xu, Yongqi Fan, Shuang-shuang Chen, Qi Ye, Weibin Guo, Xinxuan Hu

P-FOLIO: Evaluating and Improving Logical Reasoning with Abundant Human-Written Reasoning Chains (2024.findings-emnlp)

Challenge:	Accurate International Classification of Diseases (ICD) coding is crucial for hospital management and healthcare data governance.
Approach:	They propose a framework to evaluate ICD coding based on complete EMRs . they use a dataset of 560 real clinical records covering 434 common diseases .
Outcome:	The proposed framework explores the capability boundaries of large language models under different paradigms.

Copied to clipboard

Simeng Han, Aaron Yu, Rui Shen, Zhenting Qi, Martin Riddell, Wenfei Zhou, Yujie Qiao, Yilun Zhao, Semih Yavuz, Ye Liu, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Dragomir Radev, Rex Ying, Arman Cohan

Information Extraction & Text Mining

Challenge:	Existing methods on understanding the capabilities of LLMs in logical reasoning rely on binary entailment classification or synthetically derived rationales.
Approach:	They propose to annotate a human-annotated dataset consisting of diverse and complex reasoning chains for a set of realistic logical reasoning stories also written by humans.
Outcome:	The proposed model outperforms existing methods on understanding the capabilities of LLMs in logical reasoning by 10% or more.

RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation (2025.acl-long)

Copied to clipboard

Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yongkang Wu, Zhonghua Li, Ye Qi, Zhicheng Dou

MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering (2025.findings-acl)

Challenge:	Existing methods rely on separate retrievers to fetch top-k text chunks for generating evidence, and they lack joint optimization.
Approach:	They propose a framework that integrates retrieval and generation into a single, auto-regressive process, enabling LLMs to directly generate fine-grained evidence from the corpus with constrained decoding.
Outcome:	Extensive experiments on five open-domain QA datasets demonstrate the proposed framework’s superior performance across both in-domain and out-of-domain tasks.

Copied to clipboard

Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, An-Lan Wang, Chunhui Lin, Hao Feng, Zhen Zhao, Yanjie Wang, Yuliang Liu, Hao Liu, Xiang Bai, Can Huang

Multilinguality

Challenge:	Text-Centric Visual Question Answering (TEC-VQA) is a text-centric visual task understanding tool.
Approach:	They introduce a benchmark that features human expert annotations across 9 languages . they prioritize the text in question-answer pairs while disregarding visual text in images .
Outcome:	The proposed benchmarks prioritize the text in question-answer pairs while disregarding visual text in images.

VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraft (2025.emnlp-main)

Copied to clipboard

Honghao Fu, Junlong Ren, Qi Chai, Deheng Ye, Yujun Cai, Hao Wang

Experience is the Teacher: Reusing Atomic Thoughts from LLMs to Improve Medical Dialogue (2026.findings-acl)

Challenge:	Large language models (LLMs) have shown significant promise in embodied decision-making tasks within virtual open-world environments, but lack domain-specific knowledge.
Approach:	They propose a cost-effective agent framework that integrates cross-modal domain knowledge and finetunes a dedicated object detection model for visual analysis.
Outcome:	The proposed framework reduces the requirement for domain-specific training data from millions of samples to a few hundred.

Copied to clipboard

Guangya Yu, Hui Luo, Qi Ye, Ruihui Hou, Weiyan Zhang, Mingxi Shang, Xuanwu Li, ChunMing Wang, Tong Ruan

Hierarchical Document Refinement for Long-context Retrieval-augmented Generation (2025.acl-long)

Challenge:	Recent large reasoning models (LLMs) lack dynamic and diverse thinking capabilities . reusing atomic thoughts provides a practical pathway toward dynamic reasoning .
Approach:	They propose a framework that extracts atomic thoughts from teacher models and reuses them to guide reasoning and generate responses.
Outcome:	The proposed framework extracts atomic thoughts from teacher models and reuses them to guide reasoning and generate responses.

Copied to clipboard

Jiajie Jin, Xiaoxi Li, Guanting Dong, Yuyao Zhang, Yutao Zhu, Yongkang Wu, Zhonghua Li, Ye Qi, Zhicheng Dou

A Bounding Box is Worth One Token - Interleaving Layout and Text in a Large Language Model for Document Understanding (2025.findings-acl)

Challenge:	Real-world RAG applications often encounter long-context input scenarios where redundant information and noise results in higher inference costs and reduced performance.
Approach:	They propose an efficient plug-and-play refiner that leverages the structural characteristics of long documents.
Outcome:	Experiments on seven QA datasets show that LongRefiner achieves competitive performance in various scenarios while using 10x fewer computational costs and latency compared to baseline.

Copied to clipboard

Jinghui Lu, Haiyang Yu, Yanjie Wang, Yongjie Ye, Jingqun Tang, Ziwei Yang, Binghong Wu, Qi Liu, Hao Feng, Han Wang, Hao Liu, Can Huang

MulDimIF: A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models (2026.findings-acl)

Challenge:	Existing methods for integrating spatial layouts with text have limitations . existing methods produce overly long text sequences or lack autoregressive traits of LLMs .
Approach:	They introduce Interleaving Layout and Text in a Large Language Model (LayTextLLM) they use OCR-derived text and spatial layouts to integrate with LLMs for document understanding .
Outcome:	The proposed model shows an increase in performance in KIE and VQA tasks.

Copied to clipboard

Junjie Ye, Caishuang Huang, Zhuohan Chen, Wenjie Fu, Chenyuan Yang, Leyi Yang, Yilong Wu, Peng Wang, Meng Zhou, Xiaolong Yang, Tao Gui, Qi Zhang, Zhongchao Shi, Jianping Fan, Xuanjing Huang

Going “Deeper”: Structured Sememe Prediction via Transformer with Tree Attention (2022.findings-acl)

Challenge:	Existing research has focused on constraint categories, offering little guidance for improving instruction following abilities.
Approach:	They propose a multi-dimensional constraint framework that allows for instruction following . they construct 9,106 code-verifiable samples and evaluate 18 LLMs .
Outcome:	The proposed framework improves instruction following performance without compromising general performance.

Copied to clipboard

Yining Ye, Fanchao Qi, Zhiyuan Liu, Maosong Sun

Beyond Boundaries: Learning a Universal Entity Taxonomy across Datasets and Languages for Open Named Entity Recognition (2025.coling-main)

Challenge:	Existing studies ignore hierarchical structures of sememes in sememe-based semantic description systems.
Approach:	They propose a structured sememe prediction problem to predict a sememes tree with hierarchical structures rather than a set of sememas.
Outcome:	The proposed model outperforms baseline models and shows its effectiveness . it predicts a sememe tree with hierarchical structures rather than a set of sememes .

Copied to clipboard

Yuming Yang, Wantong Zhao, Caishuang Huang, Junjie Ye, Xiao Wang, Huiyuan Zheng, Yang Nan, Yuran Wang, Xueying Xu, Kaixin Huang, Yunke Zhang, Tao Gui, Qi Zhang, Xuanjing Huang

Information Extraction & Text Mining

Challenge:	Current Large Language Models struggle with complex entity taxonomies in open domains and lack NER capabilities.
Approach:	They propose a dataset to guide LLMs' generalization in Open NER under a universal entity taxonomy.
Outcome:	The proposed model outperforms GPT-4 in 3 out-of-domain benchmarks across 15 datasets and 6 languages.

UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model (2023.findings-emnlp)

Copied to clipboard

Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Lin, Fei Huang

Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric (2025.acl-long)

Challenge:	Existing studies for visually-situated language understanding have shown shallow zero-shot visual text recognition ability when fed a low-resolution image with salient text information.
Approach:	They propose a model for universal OCR-free visually-situated language understanding based on the Multimodal Large Language Model (MLLM) their model is jointly finetuned on a wide range of visually situated language understanding tasks via a unified instruction format.
Outcome:	The proposed model achieves state-of-the-art ocr-free performance in 8 out of 10 visually-situated language understanding tasks across 5 domains: documents, tables, charts, natural images, and webpage screenshots.

Copied to clipboard

Yuming Yang, Yang Nan, Junjie Ye, Shihan Dou, Xiao Wang, Shuo Li, Huijie Lv, Tao Gui, Qi Zhang, Xuanjing Huang

OneNet: A Fine-Tuning Free Framework for Few-Shot Entity Linking via Large Language Model Prompting (2024.emnlp-main)

Challenge:	Existing studies have explored various diversity-aware data selection methods to construct high-quality datasets and enhance model performance.
Approach:	They propose to use data diversity to measure instruction tuning of large language models.
Outcome:	The proposed diversity metric outperforms existing methods on simulated and real-world data and shows that it captures diversity variations and achieves a 0.97 correlation with instruction tuning.

Copied to clipboard

Xukai Liu, Ye Liu, Kai Zhang, Kehang Wang, Qi Liu, Enhong Chen

CMQCIC-Bench: A Chinese Benchmark for Evaluating Large Language Models in Medical Quality Control Indicator Calculation (2025.findings-acl)

Challenge:	Entity Linking (EL) is the process of associating ambiguous textual mentions to specific entities in a knowledge base.
Approach:	They propose a framework that utilizes the few-shot learning capabilities of Large Language Models without the need for fine-tuning to improve the accuracy of EL.
Outcome:	The framework outperforms current state-of-the-art methods in a few-shot entity linking task.

Copied to clipboard

Guangya Yu, Yanhao Li, Zongying Jiang, Yuxiong Jin, Li Dai, Yupian Lin, Ruihui Hou, Weiyan Zhang, Yongqi Fan, Qi Ye, Jingping Liu, Tong Ruan

RethinkingTMSC: An Empirical Study for Target-Oriented Multimodal Sentiment Classification (2023.findings-emnlp)

Challenge:	Medical quality control indicators are essential to assess the qualifications of healthcare institutions for medical services.
Approach:	They propose a Chinese electronic medical records-based dataset for MQCIC and propose CF-IR method that disentangles clinical fact verification and inferential rule reasoning actions.
Outcome:	The proposed method outperforms Chain-of-Thought methods on 20 representative LLMs, covering general and medical models.

Copied to clipboard

Junjie Ye, Jie Zhou, Junfeng Tian, Rui Wang, Qi Zhang, Tao Gui, Xuanjing Huang

LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing (2026.findings-acl)

Challenge:	Recent studies have shown that current TMSC systems rely on textual information, and the progress in tackling this task has slowed down.
Approach:	They propose to integrate both visual and textual information to improve the performance of TMSC by considering multimodal information.
Outcome:	The proposed model integrates both visual and textual information to improve performance.

Copied to clipboard

Hao Li, Yiqun Zhang, Zhaoyan Guo, Chenxu Wang, Shengji Tang, Qiaosheng Zhang, Yang Chen, Biqing Qi, Peng Ye, Lei Bai, Zhen Wang, Shuyue Hu

Challenge:	Large language model (LLM) routing assigns each query to the best suitable model from an ensemble.
Approach:	They introduce a large-scale benchmark and unified framework for LLM routing . they find that many routing methods exhibit similar performance under unified evaluation .
Outcome:	The proposed benchmark provides comprehensive metrics for both performance-oriented and performance-cost trade-off routing.

Hi-ArG: Exploring the Integration of Hierarchical Argumentation Graphs in Language Pretraining (2023.emnlp-main)

Copied to clipboard

Jingcong Liang, Rong Ye, Meng Han, Qi Zhang, Ruofei Lai, Xinyu Zhang, Zhao Cao, Xuanjing Huang, Zhongyu Wei

MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs (2025.findings-acl)

Challenge:	Recent studies have discussed its capability to assist language models for various applications.
Approach:	They propose a structure to organize arguments using the Hierarchical Argumentation Graph (Hi-ArG) and propose two approaches to exploit Hi-AarG, including a text-graph multi-modal model GreaseArR and a framework augmented with graph information.
Outcome:	The proposed structure supersedes existing language models on two argumentation tasks while incorporating graph information during further training improves vanilla language models.

Copied to clipboard

Yongqi Fan, Yating Wang, Guandong Wang, Zhai Jie, Jingping Liu, Qi Ye, Tong Ruan

Question Answering

Challenge:	Existing evaluation methods overlook the distinction between factoid and non-factoidic questions.
Approach:	They propose a method that distinguishes open-ended questions and ranks candidate answers . they propose QA requires longer answer statements and nuanced reasoning processes .
Outcome:	The proposed method better aligns with human annotations and offers more interpretable results.

CausalMACE: Causality Empowered Multi-Agents in Minecraft Cooperative Tasks (2025.findings-emnlp)

Copied to clipboard

Qi Chai, Zhang Zheng, Junlong Ren, Deheng Ye, Zichuan Lin, Hao Wang

Challenge:	Existing work on multi-agent collaborative tasks in Minecraft is limited due to inefficiency and limited fault tolerance.
Approach:	They propose a framework that incorporates causality to manage dependencies among subtasks.
Outcome:	The proposed framework achieves state-of-the-art performance in multi-agent cooperative tasks of Minecraft.

SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models (2026.acl-long)

Copied to clipboard

Yiyang Gu, Junwei Yang, Junyu Luo, Ye Yuan, Bin Feng, Yingce Xia, Shufang Xie, Kaili Liu, Bohan Wu, Qi Shi, Haoran Li, Beier Xiao, Zhiping Xiao, Xiao Luo, Weizhi Zhang, Philip S. Yu, Zequn Liu, Ming Zhang

ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages (2024.acl-long)

Challenge:	Existing evaluations of large language models fail to reflect fine-grained capabilities . existing benchmarks are manually curated or domain-generic, limiting scalability and alignment with real use cases.
Approach:	They propose a framework that allows custom construction of benchmarks from large-scale scientific data to evaluate application-specific scientific capabilities in LLMs.
Outcome:	The proposed framework reveals fine-grained differences in scientific capabilities that standard benchmarks overlook . it allows custom construction of benchmarks from large-scale scientific data to evaluate application-specific capabilities in LLMs.

Copied to clipboard

Junjie Ye, Sixian Li, Guanyu Li, Caishuang Huang, Songyang Gao, Yilong Wu, Qi Zhang, Tao Gui, Xuanjing Huang

Can LLMs Simulate L2-English Dialogue? An Information-Theoretic Analysis of L1-Dependent Biases (2025.acl-long)

Challenge:	Existing research focuses on enhancing LLMs capabilities through tool utilization.
Approach:	They propose a framework to investigate safety issues in large language models in tool learning . they propose malicious queries and jailbreak attacks in the input stage .
Outcome:	The proposed framework investigates six safety scenarios for LLMs in tool learning . the data will be released upon acceptance of the proposed framework .

Copied to clipboard

Rena Wei Gao, Xuetong Wu, Tatsuki Kuribayashi, Mingrui Ye, Siya Qi, Carsten Roever, Yuanxing Liu, Zheng Yuan, Jey Han Lau

Heterogeneous Graph Neural Networks for Keyphrase Generation (2021.emnlp-main)

Challenge:	Large Language Models (LLMs) can simulate non-native-like English use observed in human second language (L2) learners interfered with by their native first language (N1) knowledge.
Approach:	They use large language models to simulate non-native-like English use observed in human second language (L2) learners, and then compare their outputs to real L2 learner data.
Outcome:	The proposed models replicate L1-dependent patterns observed in human second language (L2) learners, with distinct influences from various languages.

Copied to clipboard

Jiacheng Ye, Ruijian Cai, Tao Gui, Qi Zhang

TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use (2025.findings-emnlp)

Challenge:	Existing approaches for keyphrase generation generate uncontrollable and inaccurate absent keyphrases.
Approach:	They propose a graph-based method that captures explicit knowledge from related references.
Outcome:	The proposed model improves on baseline keyphrase generation models on multiple benchmarks.

Copied to clipboard

Junjie Ye, Yilong Wu, Sixian Li, Yuming Yang, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang, Peng Wang, Zhongchao Shi, Jianping Fan, Zhengyin Du

ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios (2025.coling-main)

Challenge:	a new approach to training large language models (LLMs) overlooks task-specific characteristics in tool use, leading to performance bottlenecks.
Approach:	They propose a task-feature-based framework that mitigates the effects of suboptimal training data . they use a dataset to train large-scale LLMs and a reward mechanism tailored to error categories .
Outcome:	The proposed framework matches or surpasses open- and closed-source LLMs in tool-use performance using only 1,217 training data points.

Copied to clipboard

Junjie Ye, Guanyu Li, SongYang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Tao Ji, Qi Zhang, Tao Gui, Xuanjing Huang

UniRAG: Unified Query Understanding Method for Retrieval Augmented Generation (2025.acl-long)

Challenge:	Existing evaluations of tool learning focus on validation of tools for large language models with expected outcomes, but this focus ignores the complex capabilities required for LLMs to effectively use tools.
Approach:	They propose a fine-grained system for evaluation of large language models’ tool learning capabilities in authentic scenarios.
Outcome:	The proposed system examines seven real-world scenarios, analyzing five dimensions crucial to LLMs in tool learning: format alignment, intent comprehension, behavior planning, tool selection, and answer organization.

Copied to clipboard

Rui Li, Liyang He, Qi Liu, Zheng Zhang, Heng Yu, Yuyang Ye, Linbo Zhu, Yu Su

Tiny Scales, Great Challenges: The Limits of Multimodal LLMs in Scale Recognition (2026.acl-long)

Challenge:	Existing query augmentation methods face knowledge update lag and hallucinations in large language models (LLMs) Existing methods face two key challenges: (1) separation of query augmented and encoding tasks, which hinders information sharing and introduces cumulative errors; (2) difficulty of selecting optimal augmentation strategy for different scenarios.
Approach:	They propose a unified framework for query understanding in RAG that integrates internal and external knowledge to enhance query augmentation and encoding tasks.
Outcome:	The proposed framework outperforms traditional query augmentation methods in five knowledge-intensive benchmark tasks in both closed and open domain question answering.

Copied to clipboard

Jihang Jin, Ronghao Chen, Hao Zhang, Ziyan Liu, Huacan Wang, Qi Ye, Jingping Liu

RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning (2024.emnlp-main)

Challenge:	Existing benchmarks focus on a single type of quantity or a specific format, lacking a comprehensive evaluation of scale recognition capabilities.
Approach:	They propose a visual scale recognition benchmark built using images from COCO, Open Images, and Flickr to evaluate scale recognition capabilities of multimodal large language models.
Outcome:	The proposed model achieves 42.60% accuracy, lower than the 97.40% of humans.

Copied to clipboard

Junjie Ye, Yilong Wu, Songyang Gao, Caishuang Huang, Sixian Li, Guanyu Li, Xiaoran Fan, Qi Zhang, Tao Gui, Xuanjing Huang

Neuro-Symbolic Query Compiler (2025.findings-acl)

Challenge:	Current research emphasizes LLMs’ capacity to utilize tools in well-structured environments while overlooking their stability when confronted with the inevitable noise of the real world.
Approach:	They propose a multi-level benchmark to evaluate the robustness of large language models in tool learning by establishing five external environments with varying levels of noise.
Outcome:	The proposed model outperforms the GPT-4 model in tool learning in three critical phases: tool selection, parameter identification, and content filling.

Copied to clipboard

Yuyao Zhang, Zhicheng Dou, Xiaoxi Li, Jiajie Jin, Yongkang Wu, Zhonghua Li, Ye Qi, Ji-Rong Wen

CL2GEC: A Multi-Discipline Benchmark for Continual Learning in Chinese Literature Grammatical Error Correction (2026.acl-long)

Challenge:	Retrieval-Augmented Generation (RAG) systems are limited in their ability to process information in open-source environments.
Approach:	They propose a neuro-symbolic framework inspired by linguistic grammar rules and compiler design to formalize complex queries using a minimal yet sufficient Backus-Naur Form grammar.
Outcome:	The proposed framework is based on a backus-naur form grammar and compiler design that maintains completeness while minimizing redundancy.

Copied to clipboard

Shang Qin, Jingheng Ye, Yinghui Li, Hai-Tao Zheng, Qi Li, Jinxiao Shan, Zhixing Li, Hong-Gee Kim

Syntactic Text Processing

Challenge:	Existing CGEC benchmarks for multi-disciplinary writing are limited . continual learning (CL) is a promising solution to handle domain-specific linguistic variation and prevent catastrophic forgetting.
Approach:	They propose a Chinese Literature Continual Learning benchmark to evaluate adaptive CGEC across disciplines.
Outcome:	The proposed benchmark includes 10,000 human-annotated sentences spanning 10 disciplines, each exhibiting distinct linguistic styles and error patterns.

AutoVecCoder: Teaching LLMs to Generate Explicitly Vectorized Code (2026.findings-acl)

Copied to clipboard

ShangZhan Li, Xinyu Yin, Xuanyu Jin, Ye He, Yuxin Zhou, Yuxuan Li, Xu Han, Wanxiang Che, Qi Shi, Ting Liu, Maosong Sun

Programming Languages in NLP

Challenge:	Current development practices face a dichotomy between automation and performance.
Approach:	They propose a framework to empower LLMs with the capability of automated explicit vectorization.
Outcome:	The proposed framework achieves state-of-the-art performance on the SSE and AVX subsets of SimdBench.

RealBehavior: A Framework for Faithfully Characterizing Foundation Models’ Human-like Behavior Mechanisms (2023.findings-emnlp)

Copied to clipboard

Enyu Zhou, Rui Zheng, Zhiheng Xi, Songyang Gao, Xiaoran Fan, Zichu Fei, Jingting Ye, Tao Gui, Qi Zhang, Xuanjing Huang

Challenge:	Existing studies on human-like behaviors in foundation models do not verify their faithfulness . a simple application of psychological tools cannot faithfully characterize all human-type behaviors .
Approach:	They propose a framework to characterize humanoid behaviors in foundation models . they argue that a simple application of psychological tools cannot faithfully characterize all human-like behaviors .
Outcome:	The proposed framework assesses the faithfulness of results based on reproducibility, internal consistency, and generalizability.

Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization (2023.findings-emnlp)

Copied to clipboard

Ningyu Xu, Qi Zhang, Jingting Ye, Menghan Zhang, Xuanjing Huang

Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levels (2025.emnlp-main)

Challenge:	Large language models (LLMs) have implicitly transfer knowledge across languages, but not all languages have such generalization capabilities.
Approach:	They propose a meta-learning-based method to learn to align conceptual spaces of different languages to enhance cross-lingual generalization.
Outcome:	The proposed method achieves competitive results with state-of-the-art methods and narrows the performance gap between languages.

Copied to clipboard

Junjie Ye, Yuming Yang, Yang Nan, Shuo Li, Qi Zhang, Tao Gui, Xuanjing Huang, Peng Wang, Zhongchao Shi, Jianping Fan

VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training (2026.acl-long)

Challenge:	Large language models (LLMs) acquire substantial world knowledge during pretraining, which is further shaped by post-training techniques such as supervised fine-tuning (SFT).
Approach:	They evaluate closed-book question answering (CBQA) performance across five LLMs from the LLaMA-2 and LLama-3 families and examine the impact of supervised fine-tuning on model knowledge.
Outcome:	The proposed model performance is 14% worse than models fine-tuned on 1,920 samples and 12% worse on 240 samples.

Copied to clipboard

Dingwei Zhu, Shihan Dou, Zhiheng Xi, Senjie Jin, Guoqiang Zhang, Jiazheng Zhang, Junjie Ye, Mingxu Chai, Enyu Zhou, Ming Zhang, Yuhui Wang, Caishuang Huang, Chenhao Huang, Yunke Zhang, Yuran Wang, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang

Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments (2026.findings-acl)

Challenge:	Reinforcement Learning (RL) in real-world environments often suffers from ambiguous or incomplete supervision.
Approach:	They propose a framework that enhances value modeling for robust RL in LLM post-training by integrating auxiliary losses guided by entropy and perplexity from a frozen language model and variational information bottleneck.
Outcome:	The proposed framework outperforms baselines on multi-turn dialogue, math reasoning, and science QA with rule-based and model-based rewards.

Copied to clipboard

Junjie Ye, Changhao Jiang, Zhengyin Du, Yufei Xu, Xuesong Yao, Zhiheng Xi, Xiaoran Fan, Qi Zhang, Tao Gui, Xuanjing Huang, Jiecao Chen

What Makes a Good Order of Examples in In-Context Learning (2024.findings-acl)

Challenge:	Currently, there are no efficient reinforcement learning (RL) frameworks specifically designed for tool use.
Approach:	They propose an automated environment construction pipeline that incorporates scenario decomposition, document generation, function integration, complexity scaling, and localized deployment to enable high-quality training environments without external tools.
Outcome:	The proposed framework significantly improves the models’ tool-use performance without degrading their general capabilities.

Copied to clipboard

Qi Guo, Leiyu Wang, Yidong Wang, Wei Ye, Shikun Zhang

Responsible & Trustworthy NLP

Challenge:	Large language models (LLMs) demonstrate impressive few-shot learning capabilities via in-context learning (ICL).
Approach:	They propose to use unlabeled data to evaluate order performance . they propose to filter out subsets of orders with label fairness and select the most influential order for each test instance.
Outcome:	The proposed method is superior over strong baselines and validates generalizability across settings.

Uncertainty-Aware Label Refinement for Sequence Labeling (2020.emnlp-main)

Copied to clipboard

Tao Gui, Jiacheng Ye, Qi Zhang, Zhengyan Li, Zichu Fei, Yeyun Gong, Xuanjing Huang

Green & Sustainable NLP

Challenge:	Conditional random fields (CRF) for label decoding have been a problem for many tasks.
Approach:	They propose a two-stage label decoding framework that model long-term label dependencies while being much more computationally efficient.
Outcome:	The proposed method outperforms the CRF-based methods and greatly accelerates the inference process.

ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use (2025.acl-long)

Copied to clipboard

Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu, Zhiheng Xi, Siyu Yuan, Tao Gui, Qi Zhang, Xuanjing Huang, Jiecao Chen

Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer? (2022.emnlp-main)

Challenge:	Effective evaluation of multi-hop tool use is critical for analyzing the understanding, reasoning, and function-calling capabilities of large language models.
Approach:	They propose a dataset that provides rigorous evaluation of multi-hop tool use.
Outcome:	The proposed model achieves 49.04% accuracy across five model families.

Copied to clipboard

Ningyu Xu, Tao Gui, Ruotian Ma, Qi Zhang, Jingting Ye, Menghan Zhang, Xuanjing Huang

TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing (2021.acl-demo)

Challenge:	Multilingual BERT (mBERT) has demonstrated considerable cross-lingual syntactic ability, but it is not well understood what leads to this variation and whether it fairly reflects difference between languages.
Approach:	They propose to use multilingual BERT to enable zero-shot cross-lingual transfer of syntactic knowledge between different languages by generating grammatical relations in 24 different languages.
Outcome:	The results show that the distance between the distributions of different languages is highly consistent with the syntactic difference in terms of linguistic formalisms.

Copied to clipboard

Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, Qinzhuo Wu, Zhengyan Li, Chong Zhang, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin, Bolin Zhu, Xiaoyu Xing, Jinlan Fu, Yue Zhang, Minlong Peng, Xiaoqing Zheng, Yaqian Zhou, Zhongyu Wei, Xipeng Qiu, Xuanjing Huang

One2Set: Generating Diverse Keyphrases as a Set (2021.acl-long)

Challenge:	Existing approaches to textual robustness evaluation focus on slightly modifying the input data, which maintains the original meaning and results in a different prediction.
Approach:	They propose a multilingual robustness evaluation toolkit for NLP that integrates universal text transformations, task-specific transformations and adversarial attack.
Outcome:	The toolkit includes universal text transformation, task-specific transformation, adversarial attack, subpopulation, and their combinations to provide comprehensive robustness analyses.

Copied to clipboard

Jiacheng Ye, Tao Gui, Yichao Luo, Yige Xu, Qi Zhang

Challenge:	Recent keyphrase generation models are wrongly imposing a predefined order on keyphrases . a new training paradigm is proposed to concatenate keyphrase sequences in parallel .
Approach:	They propose a training paradigm that concatenates keyphrases in a predefined order . they propose combining a fixed set of learned control codes with a bipartite matching mechanism .
Outcome:	The proposed model outperforms the state-of-the-art methods on multiple benchmarks.

Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training (2026.acl-long)

Copied to clipboard

Changhao Jiang, Ming Zhang, Yifei Cao, Junjie Ye, Xiaoran Fan, Shihan Dou, Zhiheng Xi, Jiajun Sun, Yi Dong, Yujiong Shen, Jingqi Tong, Baoyu Fan, Tao Gui, Qi Zhang, Xuanjing Huang

When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation? (N18-2)

Challenge:	Existing methods to predict performance of large language models are lacking . authors propose a size-dependent mutual information predictor for closed-book question answering accuracy .
Approach:	They propose a size-dependent mutual information predictor that integrates knowledge frequency, knowledge specificity, and model size to forecast closed-book question answering accuracy.
Outcome:	The proposed method outperforms baseline models and achieves R2 > 0.7 in predicting QA accuracy without additional training.

Copied to clipboard

Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, Graham Neubig