Papers by Shen Ma
Mention Extraction and Linking for SQL Query Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing text-to-SQL systems take a slot-filling approach, but they are limited in capturing inter-dependencies among SQL clauses. |
| Approach: | They propose an extraction-linking approach where a unified extractor recognizes all types of slot mentions appearing in the question sentence before a linker maps the recognized columns to the table schema to generate executable SQL queries. |
| Outcome: | The proposed method achieves the first place on the WikiSQL benchmark. |
Learning from Mistakes: Negative Reasoning Samples Enhance Out-of-Domain Generalization (2026.acl-long)
Copied to clipboard
Tian Xueyun, MingHua Ma, Bingbing Xu, Nuoyan Lyu, Wei Li, Heng Dong, Zheng Chu, Yuanzhuo Wang, Huawei Shen
| Challenge: | Recent studies show that supervised fine-tuning (SFT) is a common approach for reasoning in large language models. |
| Approach: | They propose to use supervised fine-tuning (SFT) on chain-of-thought trajectories demonstrations . they find that incorporating negative traxories yields substantial OOD generalization gains . |
| Outcome: | The proposed scheme yields 5.51% OOD gain over positive-only training. |
Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are proving significant potential in healthcare, prompting numerous benchmarks to evaluate their capabilities. |
| Approach: | They propose a framework that deconstructs benchmark development into five stages from design to governance and provides a checklist of 46 medically-tailored criteria. |
| Outcome: | The framework deconstructs benchmark development into five stages from design to governance and provides a comprehensive checklist of 46 medically-tailored criteria. |
Towards Real-World Writing Assistance: A Chinese Character Checking Benchmark with Faked and Misspelled Characters (2024.acl-long)
Copied to clipboard
Yinghui Li, Zishan Xu, Shaoshen Chen, Haojing Huang, Yangning Li, Shirong Ma, Yong Jiang, Zhongli Li, Qingyu Zhou, Hai-Tao Zheng, Ying Shen
| Challenge: | Existing studies focus on misspelled characters, ignoring faked characters which are more common and difficult to correct. |
| Approach: | They propose to use Chinese character checking to identify and correct wrong characters in texts by human annotation. |
| Outcome: | The proposed dataset is the first real-world visual and the largest human-crafted dataset for the Chinese character checking scenario. |
Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents (2026.findings-acl)
Copied to clipboard
Ying He, Zhouhong Gu, Zhecheng Hu, Yubo Zhou, Hao Shen, Jiaqing Liang, Zhaoqian Dai, Ma Shuguang, Fei Yu, Yanghua Xiao, Zhixu Li
| Challenge: | Existing LLMs struggle to identify errors in financial documents, a study shows . 18% of financial practitioners make errors daily, one-third make errors several times weekly, and 59% make errors multiple times monthly. |
| Approach: | They introduce FinED-Bench, a publicly available Benchmark for financial error detection . it covers nine real-world financial scenarios and includes over 900 documents in 2025 . supervised fine-tuning can significantly improve the performance of weaker LLMs, they show . |
| Outcome: | The proposed benchmark covers nine real-world financial scenarios and includes over 900 documents reported in 2025 that are unseen by existing language models. |
ACR: Adaptive Context Refactoring via Context Refactoring Operators for Multi-Turn Dialogue (2026.findings-acl)
Copied to clipboard
Jiawei Shen, Jia Zhu, Hanghui Guo, Weijie Shi, Yue Cui, Qingyu Niu, Guoqing Ma, Jingjiang Liu, Yidan Liang, Yilin Wang, Shimin Di, Jiajie Xu
| Challenge: | Existing approaches to multi-turn dialogues lack contextual consistency and dependencies, and models struggle to maintain factual faithfulness as interaction turns increase. |
| Approach: | They propose an adaptive context refactoring framework that monitors and reshapes the interaction history to mitigate contextual inertia and state drift. |
| Outcome: | The proposed model outperforms baselines while reducing token consumption. |
An Optimizable Suffix Is Worth A Thousand Templates: Efficient Black-box Jailbreaking without Affirmative Phrases via LLM as Optimizer (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing jailbreaking methods generate harmful and unethical content when subjected to jailbreaking attacks. |
| Approach: | They propose a black-box jailbreaking method with optimizable suffixes that translate jailbreaking objectives into natural language instructions. |
| Outcome: | The proposed method outperforms existing methods by 2.4 times in the ASR of three open-source LLMs and GPT-3.5-Turbo. |
The Death and Life of Great Prompts: Analyzing the Evolution of LLM Prompts from the Structural Perspective (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has shown that high-quality prompts are essential for LLMs to produce accurate and relevant responses. |
| Approach: | They analyze 10,538 in-the-wild prompts collected from various platforms and develop a framework that decomposes the prompts into eight key components. |
| Outcome: | The proposed framework decomposes 10,538 in-the-wild prompts into eight components. |
SynET: Synonym Expansion using Transitivity (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to find synonyms from text corpora are distributed and pattern based, but they suffer from low precision and low recall. |
| Approach: | They propose a task of synonym expansion using transitivity and propose auxiliary task to reduce the impact of noisy sentences. |
| Outcome: | The proposed approach reduces the impact of noisy sentences and reduces noise in a real-world dataset. |
DB-Explore: Automated Database Exploration and Instruction Synthesis for Text-to-SQL (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent text-to-SQL systems that use large language models struggle with complex database structures and domain-specific queries. |
| Approach: | a framework that aligns large language models with database knowledge is proposed . DB-Explore constructs database graphs to capture complex relational schemas . |
| Outcome: | a new framework outperforms existing text-to-SQL systems by outperforming existing systems. |
LLM Agents in Law: Taxonomy, Applications, and Challenges (2026.acl-long)
Copied to clipboard
Shuang Liu, Ruijia Zhang, Ruoyun Ma, Yujia Deng, Lanyi Zhu, Jiayu Li, Zelong Li, Zhibin Shen, Mengnan Du
| Challenge: | Large language models (LLMs) have improved the legal domain, but deployment of standalone models faces significant limitations regarding hallucination, outdated information, and verifiability. |
| Approach: | They present a survey of LLM agents for legal tasks and analyze their architectures . they analyze the transition from standard legal LLMs to legal agents . |
| Outcome: | The proposed architectures bridge the gap between technical capabilities and domain-specific needs. |
CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating the performance of Grammatical Error Correction systems is a challenging task due to its subjectivity. |
| Approach: | They propose a method to evaluate GEC systems in multi-reference evaluation setting . they use consistent edit boundaries to eliminate bias caused by inconsistent edit boundaries . |
| Outcome: | The proposed evaluation metric eliminates bias caused by inconsistent edit boundaries on six English reference sets. |
The Mirage of Model Editing: Revisiting Evaluation in the Wild (2025.acl-long)
Copied to clipboard
| Challenge: | despite near-perfect results, effectiveness of model editing in real-world applications remains unclear. |
| Approach: | They propose QAEdit and WILD to better reflect real-world use of model editing . they propose a benchmark aligned with widely used question answering datasets and a task-agnostic evaluation framework . |
| Outcome: | The proposed QAEdit benchmark and WILD evaluation framework show that current models perform worse than previously reported. |
Pause or Fabricate? Training Language Models for Grounded Reasoning (2026.findings-acl)
Copied to clipboard
Yiwen Qiu, Linjuan Wu, Yizhou Liu, Yuchen Yan, Jin Ma, Xu Tan, Yao Hu, Daoxin Zhang, Wenqi Zhang, Weiming Lu, Jun Xiao, Yongliang Shen
| Challenge: | Large language models implicitly fabricate information when inputs are incomplete, causing confidence but unreliable conclusions. |
| Approach: | They propose a framework for grounded reasoning under incomplete information that decomposes reasoning into two stages . they propose stage-specific rewards to penalize hallucinations, enabling models to detect gaps, stop proactively, and resume reasoning after clarification. |
| Outcome: | The proposed framework improves premise detection and task success by 30% . it also reduces average response length by over 20% . |
Leveraging Outline-Optimized Generative Interactions and Critique for Self-Refining Outlines with Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Logic-RL is a framework that transforms critique-guided outline refinement into a learnable policy through reinforcement learning. |
| Approach: | They propose a framework that transforms critique-guided outline refinement into a learnable policy through reinforcement learning. |
| Outcome: | The proposed framework improves on FreshWiki and WikiOutline . it can be iteratively applied, with improved quality continuing through three refinement rounds before diminishing returns. |
Automated Fine-Grained Mixture-of-Experts Quantization (2025.findings-acl)
Copied to clipboard
| Challenge: | specialized quantization framework for Mixture of Experts architectures is inadequate for model compression. |
| Approach: | They propose a specialized quantization framework for Mixture of Experts architectures . they find that expert networks exhibit distinctive channel-wise outlier distributions ." |
| Outcome: | The proposed framework improves on the Mixtral-8x7b-v0.1 architecture while maintaining minimal computational overhead. |
Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding (2025.acl-short)
Copied to clipboard
| Challenge: | Current solutions incur prohibitive training costs, leaving statistical behaviors and cost-effective approaches underexplored. |
| Approach: | They propose a positional contrast decoding technique that contrasts long-aware attention with designed local-awn attention. |
| Outcome: | The proposed model achieves state-of-the-art performance on long-context benchmarks. |
PFDial: A Structured Dialogue Instruction Fine-tuning Method Based on UML Flowcharts (2025.findings-acl)
Copied to clipboard
Ming Zhang, Yuhui Wang, Yujiong Shen, Tingyi Yang, Changhao Jiang, Yilong Wu, Shihan Dou, Qinhao Chen, Zhiheng Xi, Zhihao Zhang, Yi Dong, Zhen Wang, Zhihui Fei, Mingyang Wan, Tao Liang, Guojun Ma, Qi Zhang, Tao Gui, Xuanjing Huang
| Challenge: | Large Language Models (LLMs) have shown remarkable progress in dialogue and reasoning, but they struggle to solve strictly constrained dialogue tasks. |
| Approach: | They construct a dataset that contains 12,705 high-quality Chinese dialogue instructions from 440 flowcharts containing 5,055 process nodes. |
| Outcome: | The proposed model outperforms GPT-4o models on backward transitions and outperformed GPT-42 models on the same dataset. |
FACTrial: Factorized Clinical Contrastive Training for Scalable Patient-Trial Retrieval (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to patient–trial retrieval rely on generic semantic matching and zero-shot transfer. |
| Approach: | They propose a factorized contrastive training framework that synthesizes diagnosis-aware supervision for scalable patient–trial retrieval. |
| Outcome: | Experiments show that the proposed framework improves quality and recall coverage. |
ParaCook: On Time-Efficient Planning for Multi-Agent Systems (2026.findings-acl)
Copied to clipboard
Shiqi Zhang, Xinbei Ma, Yunqing Xu, Zouying Cao, Pengrui Lu, Haobo Yuan, Tiancheng Shen, Zhuosheng Zhang, Hai Zhao, Ming-Hsuan Yang
| Challenge: | Existing agent benchmarks focus on task completion while neglecting time efficiency in parallel and asynchronous operations. |
| Approach: | They propose a framework for large language models that allows agents to plan long-horizon tasks in a scalable way. |
| Outcome: | The proposed framework is based on the Overcooked game and can be used to evaluate time efficiency-aware multi-agent planning. |
Logic: Long-form Outline Generation via Imitative and Critical Self-refinement (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for long-form outline generation have low knowledge density and lack detail . retrieval-augmented approaches struggle to maintain logical coherence across retrieved information . |
| Approach: | They propose a system that mimics human writers' refinement process by mimicking outlines through imitation and critical self-refinement. |
| Outcome: | The proposed system improves on the FreshWiki and WikiOutline datasets and establishes a coherent planning framework and structured knowledge base. |
Mixture of Heterogeneous Grouped Experts for Language Modeling (2026.acl-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) based on Mixture-of-Experts (MoE) enforce uniform expert sizes, creating a rigidity that fails to align computational costs with varying token-level complexity. |
| Approach: | They propose a mixture of heterogeneous grouped experts (MoHGE) that allows for flexible, resource-aware expert combinations. |
| Outcome: | The proposed model matches the performance of existing Mixture-of-Experts architectures while maintaining balanced GPU utilization. |
Unifying Language Agent Algorithms with Graph-based Orchestration Engine for Reproducible Agent Research (2025.acl-demo)
Copied to clipboard
Qianqian Zhang, Jiajia Liao, Heting Ying, Yibo Ma, Haozhan Shen, Jingcheng Li, Peng Liu, Lu Zhang, Chunxin Fang, Kyusong Lee, Ruochen Xu, Tiancheng Zhao
| Challenge: | Language agents powered by large language models (LLMs) have demonstrated remarkable capabilities in understanding, reasoning, and executing complex tasks. |
| Approach: | They propose a flexible framework that addresses engineering overhead and insufficient evaluation frameworks for fair comparison. |
| Outcome: | The proposed framework simplifies language agent development and establishes a foundation for reproducible agent research. |
MKT: A Multi-Stage Knowledge Transfer Framework to Mitigate Catastrophic Forgetting in Multi-Domain Chinese Spelling Correction (2025.emnlp-industry)
Copied to clipboard
Peng Xing, Yinghui Li, Shirong Ma, Xinnian Liang, Haojing Huang, Yangning Li, Shu-Yu Guo, Hai-Tao Zheng, Wenhao Jiang, Ying Shen
| Challenge: | Chinese Spelling Correction (CSC) is a model that detects and corrects spelling errors in given sentences. |
| Approach: | They propose a model-agnostic model with an evolving teacher model and dynamic distillation weights for knowledge transfer in each domain rather than focusing solely on new domain knowledge. |
| Outcome: | The proposed model-agnostic framework is based on an evolving teacher model and dynamic distillation weights for knowledge transfer in each domain, rather than focusing solely on new domain knowledge. |
Entailment Tree Explanations via Iterative Retrieval-Generation Reasoner (2022.findings-naacl)
Copied to clipboard
Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, Rui Dong, Xiaokai Wei, Henghui Zhu, Xinchi Chen, Peng Xu, Zhiheng Huang, Andrew Arnold, Dan Roth
| Challenge: | Large language models have achieved high performance on various natural language benchmarks, but the explainability of their output remains elusive. |
| Approach: | They propose an architecture called iterative retrieval-generation reasoner that generates an entailment tree that explains a given hypothesis by using premises from C. |
| Outcome: | The proposed model outperforms existing benchmarks on premise retrieval and entailment tree generation with around 300% gain in overall correctness. |
PLAWBENCH: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice (2026.acl-long)
Copied to clipboard
Yuzhen Shi, Huanghai Liu, Yiran HU, Song Gaojie, Xu Xinran, Yubo Ma, Tianyi Tang, Li Zhang, Qingjing Chen, Feng Di, Wenbo Lv, Weiheng Wu, Kexin Yang, Sen Yang, Wei Wang, Rongyao Shi, Qiu Yuanyang, Yuemeng Qi, Zhang Jingwen, Sui Xiaoyu, Yifan Chen, Zhang Yi, An Yang, Bowen Yu, Dayiheng Liu, Junyang Lin, Weixing Shen, Bing Zhao, Charles L. A. Clarke, HU Wei
| Challenge: | Existing benchmarks for large language models (LLMs) are coarse, single-dimensional metrics and do not explicitly assess fine-grained legal reasoning. |
| Approach: | They propose a Practical Law Benchmark to evaluate large language models in real-world legal practice scenarios. |
| Outcome: | The proposed model is based on 850 questions and 13 scenarios with expert-designed evaluation rubrics. |
False Friends in the Shell: Unveiling the Emoticon Semantic Confusion in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Emoticons are widely used in digital communication to convey affective intent, yet their safety implications for Large Language Models (LLMs) remain largely unexplored. |
| Approach: | They propose to use ASCII-based emoticons to perform unintended actions in large language models (LLMs) This vulnerability is pervasive, with an average confusion ratio exceeding 38%, and 90% of confused responses yield 'silent failures' authors call on the community to recognize this emerging vulnerability and develop effective mitigation methods to uphold the safety and reliability of human-LLM interactions. |
| Outcome: | The proposed framework exploits emoticon semantic confusion in six LLMs and demonstrates that existing prompt-based mitigations are ineffective. |
Too Good to be Bad: On the Failure of LLMs to Role-Play Villains (2026.findings-acl)
Copied to clipboard
Zihao Yi, Qingxuan Jiang, Ruotian Ma, Xingyu Chen, Qu Yang, Mengru Wang, Fanghua Ye, Ying Shen, Zhaopeng Tu, Xiaolong Li, Liefeng Bo
| Challenge: | Large Language Models (LLMs) are increasingly tasked with creative generation, but their ability to portray non-prosocial, antagonistic personas remains largely unexamined. |
| Approach: | They propose a moral alignment benchmark to test the safety of large language models . they find that models struggle with traits directly antithetical to safety principles . |
| Outcome: | The proposed model fails to accurately portray morally ambiguous or villainous characters . the model fails most with traits directly antithetical to safety principles . |
Lost in Pronunciation: Detecting Chinese Offensive Language Disguised by Phonetic Cloaking Replacement (2025.emnlp-industry)
Copied to clipboard
Haotan Guo, Jianfei He, Jiayuan Ma, Hongbin Na, Zimu Wang, Haiyang Zhang, Qi Chen, Wei Wang, Zijing Shi, Tao Shen, Ling Chen
| Challenge: | Phonetic Cloaking Replacement (PCR) is a problem in content moderation in China. |
| Approach: | They organize PCR into a four-way surface-form taxonomy and compile PCR-ToxiCN, a dataset of 500 phonetically cloaked offensive posts gathered from the RedNote platform. |
| Outcome: | The proposed model achieves only an F1-score and zero-shot chain-of-thought prompting pushes performance even lower. |
CL-QR: Cross-Lingual Enhanced Query Reformulation for Multi-lingual Conversational AI Agents (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Existing QR systems that reformulate defective user queries are limited in English due to the scarcity of non-English QR labels. |
| Approach: | They propose a query reformulation method which reformulates defective user queries to improve non-English QR performance. |
| Outcome: | The proposed framework improves non-English QR performance by leveraging abundant reformulation resources in English. |
DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimization (2026.findings-eacl)
Copied to clipboard
Haiyang Shen, Hang Yan, Zhongshi Xing, Mugeng Liu, Yue Li, Zhiyang Chen, Yuxiang Wang, Jiuzheng Wang, Yun Ma
| Challenge: | Existing retrieval-augmented generation paradigms rely heavily on public knowledge . Existing RAGs reliant on public information and often falter when faced with domain-specific queries. |
| Approach: | They propose a framework that combines a data-construction modeling approach with a scalable synthetic data-generation pipeline to optimize domain-specific retrieval performance. |
| Outcome: | The proposed framework optimizes domain-specific retrieval performance and bolsters retriever robustness. |
MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains (2025.findings-naacl)
Copied to clipboard
Guoli Yin, Haoping Bai, Shuang Ma, Feng Nan, Yanchao Sun, Zhaoyang Xu, Shen Ma, Jiarui Lu, Xiang Kong, Aonan Zhang, Dian Ang Yap, Yizhe Zhang, Karsten Ahnert, Vik Kamath, Mathias Berglund, Dominic Walsh, Tobias Gindele, Juergen Wiest, Zhengfeng Lai, Xiaoming Simon Wang, Jiulong Shan, Meng Cao, Ruoming Pang, Zirui Wang
| Challenge: | Existing benchmarks focus on specific application scenarios, emphasizing task completion but failing to dissect the underlying skills that drive these outcomes. |
| Approach: | They propose a Massive Multitask Agent Understanding benchmark that evaluates LLMs across five domains and offline tasks. |
| Outcome: | The Massive Multitask Agent Understanding (MMAU) benchmark evaluates models across five domains including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. |
Natural-Language Policies to Executable Decisions: An Interpretable Large Language Model Framework (2026.acl-industry)
Copied to clipboard
Ziqiang Zhang, Jing Ma, Zilong Wang, Jiayuan Chen, Yi Qiao, Yu He, Wei Zhang, Dai Cheng, Xiaoyu Shen
| Challenge: | a production-grade pricing system for tourism is challenging due to unstructured nature of travel orders and ever-evolving pricing policies. |
| Approach: | They propose a production-grade pricing system with a strict decision boundary . they propose to combine structured extraction and bounded policy/path selection with interpretable condition trees . |
| Outcome: | The proposed system processed 3,960 orders in six months and reduced the order management team from 15-20 to 3 . the system reduced the per-order handling time from 10 minutes to 2 minutes. |
Source-free Domain Adaptation for Aspect-based Sentiment Analysis (2024.lrec-main)
Copied to clipboard
| Challenge: | Unsupervised Domain Adaptation (UDA) of the Aspect-based Sentiment Analysis task is a data mining technique that involves aspect extraction and aspect sentiment classification subtasks. |
| Approach: | They propose a framework that allows model parameter transfer, not data transfer, between different domains. |
| Outcome: | The proposed framework performs competitively with traditional unsupervised domain adaptation methods under privacy conditions. |
SQL Generation via Machine Reading Comprehension (2020.coling-main)
Copied to clipboard
| Challenge: | Text-to-SQL systems can generate SQL queries given natural language questions. |
| Approach: | They propose a method that formulates a question answering problem as a query answering problem where different slots are predicted by a unified machine reading comprehension (MRC) model. |
| Outcome: | The proposed method can achieve competitive results on WikiSQL, suggesting it being a promising direction for text-to-SQl. |
On the (In)Effectiveness of Images for Text Classification (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing studies have focused on text classification, but have shown that images do not improve NLP tasks. |
| Approach: | They focus on text classification, where images complement the text and the Wikipedia page can be in one of a number of different languages. |
| Outcome: | The proposed model trains without external pre-training, but when combined with BERT models pre-trained on large-scale external data, images contribute nothing. |
Multi-View Reasoning: Consistent Contrastive Learning for Math Word Problem (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating complex semantics and diverse equations are limited by a fixed view. |
| Approach: | They propose a multi-view consistent contrastive learning approach that decouples human reasoning into two independent but consistent views. |
| Outcome: | The proposed approach significantly outperforms existing baselines on complex problems on multiple languages. |
CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalization (2026.acl-long)
Copied to clipboard
Zhongyuan Peng, Yifan Yao, Kaijing Ma, Shuyue Guo, Yizhe Li, Yichi Zhang, Chenchen Zhang, Yifan Zhang, Zhouliang Yu, Luming Li, Minghao Liu, Yihang Xia, Jiawei Shen, Yuchen Wu, Yixin Cao, Zhaoxiang Zhang, Wenhao Huang, Jiaheng Liu, Ge Zhang
| Challenge: | Existing approaches to formalizing mathematical statements face limitations in accuracy, especially in the context of complex, highlevel problems that involve sophisticated mathematical reasoning. |
| Approach: | They propose a CriticLean framework that elevates the role of the critic from a passive validator to an active learning component and introduce a benchmark to measure models’ ability to distinguish semantically correct from incorrect formalizations. |
| Outcome: | The proposed framework outperforms open- and closed-source benchmarks and shows that it significantly outperformed existing models. |
ChatMusician: Understanding and Generating Music Intrinsically with LLM (2024.findings-acl)
Copied to clipboard
Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, Liumeng Xue, Ziyang Ma, Qin Liu, Tianyu Zheng, Yizhi Li, Yinghao Ma, Yiming Liang, Xiaowei Chi, Ruibo Liu, Zili Wang, Chenghua Lin, Qifeng Liu, Tao Jiang, Wenhao Huang, Wenhu Chen, Jie Fu, Emmanouil Benetos, Gus Xia, Roger Dannenberg, Wei Xue, Shiyin Kang, Yike Guo
| Challenge: | Despite LLMs' impressive capabilities in musical knowledge, music reasoning remains an unsolved task. |
| Approach: | They propose an open-source large language model (LLM) that integrates intrinsic musical abilities into LLaMA2 and GPT-3.5. |
| Outcome: | The proposed model can understand and generate music with a pure text tokenizer without external multi-modal neural structures or tokenizers. |
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. |
| Approach: | They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them. |
| Outcome: | The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods. |
Re-Reading Improves Reasoning in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Unlike thought-eliciting prompting methods, RE2 shifts the focus to the input by processing questions twice, thereby enhancing the understanding process. |
| Approach: | They introduce a simple, yet general and effective prompting method, RE2, which rereads the question as input. |
| Outcome: | The proposed method demonstrates strong generality and compatibility with most thought-eliciting prompting methods, including CoT. |
UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization (2026.acl-long)
Copied to clipboard
Zhengxi Lu, Fei Tang, Guangyi Liu, Jin Ma, Kaitao Song, Xu Tan, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
| Challenge: | Experimental results show that UI-Copilot-7B achieves state-of-the-art performance on challenging MemGUI-Bench, outperforming strong 7B-scale GUI agents such as GUI-Owl-7B and UITARS-1.5-7B. |
| Approach: | They propose a collaborative framework where the GUI agent focuses on task execution while a lightweight copilot provides on-demand assistance for memory retrieval and numerical computation. |
| Outcome: | The proposed framework outperforms GUI-Owl-7B and UI-TARS-1.5-7B on MemGUI-Bench and delivers 17.1% improvement on AndroidWorld over the base Qwen model. |
ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities (2025.findings-naacl)
Copied to clipboard
Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, Ruoming Pang
| Challenge: | Recent advances in large language models have led to a growing interest in tool assisted LLMs . toolSandbox includes stateful tool execution, implicit state dependencies between tools . |
| Approach: | a new tool-based evaluation tool is released to help LLMs evaluate their tool-use capabilities. a tool-driven evaluation tool includes stateful tool execution, implicit state dependencies between tools and a built-in user simulator. |
| Outcome: | the toolSandbox evaluation benchmark shows that open source and proprietary models have a performance gap . the benchmarks show that even the most capable LLMs are challenged by state dependent tasks . |
An Expression Tree Decoding Strategy for Mathematical Equation Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to generate mathematical equations from natural language ignore parallel or dependent relations between math expressions. |
| Approach: | They propose to integrate tree structure into the expression-level generation and advocate an expression tree decoding strategy. |
| Outcome: | The proposed method outperforms baseline methods for generating mathematical equations from natural language. |
Traces in the Brain: Neural Evidence for Syntactic Movement in English and Chinese (2026.findings-acl)
Copied to clipboard
Yuhan Huang, Zhengwu Ma, Yuqi Jin, Beth Chan, Zheng Shen, Jackie Yan-Ki Lai, John T. Hale, Jixing Li
| Challenge: | Syntactic movement is a core concept in generative linguistics to account for word-order variation and long-distance dependencies. |
| Approach: | They annotated every sentence in the audiobook The Little Prince using X-bar style tree annotations. |
| Outcome: | The proposed model shows that deep structure significantly predicts neural responses in English but not in Chinese. |
PEAR: Planner-Executor Agent Robustness Benchmark (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies examine isolated attack surfaces or specific scenarios, leaving a lack of holistic understanding of MAS vulnerabilities. |
| Approach: | They propose a benchmark to evaluate the utility and vulnerability of planner–executor MAS. |
| Outcome: | The proposed benchmark evaluates planner–executor MAS on a widely adopted design. |
RSDA: Restoring Stale Data Affinity via Dynamic Renovation Strategy for Mitigating Data Scarcity (2026.acl-long)
Copied to clipboard
Yidan Liang, Jia Zhu, Weijie Shi, Hanghui Guo, Yue Cui, Jiawei Shen, Guoqing Ma, Jingjiang Liu, Qingyu Niu, Yilin Wang, Shimin Di, Jiajie Xu
| Challenge: | High-quality data is the cornerstone of advancing large language models, but the supply of premium data is nearing depletion, while vast stale corpora remain underutilized. |
| Approach: | They propose a framework to restore stale data affinity by quantifying the latent value of samples and employing a dynamic renovation strategy selection mechanism to determine the optimal component-level strategy. |
| Outcome: | The proposed framework achieves performance improvements using less than 10% of the data volume, underscoring that the latent potential of stale corpora remains largely untapped. |
Linguistic Rules-Based Corpus Generation for Native Chinese Grammatical Error Correction (2022.findings-emnlp)
Copied to clipboard
Shirong Ma, Yinghui Li, Rongyi Sun, Qingyu Zhou, Shulin Huang, Ding Zhang, Li Yangning, Ruiyang Liu, Zhongli Li, Yunbo Cao, Haitao Zheng, Ying Shen
| Challenge: | Chinese Grammatical Error Correction (CGEC) is a challenging NLP task and a common application in human daily life. |
| Approach: | They propose a linguistic rules-based approach to construct large-scale CGEC training corpora with automatically generated grammatical errors. |
| Outcome: | The proposed method improves performance of existing CGEC models and the benchmark is excellent resource for further development. |
A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences (2025.findings-acl)
Copied to clipboard
Jiaxin Shen, Jinan Xu, Huiqi Hu, Luyi Lin, Guoyang Ma, Fei Zheng, Fandong Meng, Jie Zhou, Wenjuan Han
| Challenge: | a recent study focuses on generating impartial and interpretable judicial judgments based on established criminal fact. |
| Approach: | They propose a law reasoning schema enriched with hierarchical factum probandum, evidence, and implicit experience that enables public scrutiny and preventing bias. |
| Outcome: | The proposed schema enables public scrutiny and prevents bias in the "Intelligent Court" it employs a suite of legal analysis tools to address the challenge task. |
Can Reasoning Path still be Effective as Input? Bridging Post-Reasoning to Chain-of-Thought Compression (2026.acl-long)
Copied to clipboard
Chengzhengxu Li, Xiaoming Liu, Zhaohan Zhang, Shengchao Liu, Guoxin Ma, Yu Lan, Cong Wang, Chao Shen
| Challenge: | Existing work on reducing CoT generation in reasoning impairs the necessary information for deriving the correct answer. |
| Approach: | They propose a reasoning paradigm that takes CoT as a part of context to simplify the reasoning task for Large Language Models (LLMs). |
| Outcome: | The proposed framework reduces the generation length of LLMs, but its effectiveness hinges on the efficiency and reliability of the contextual CoT generation. |
FASTMATCH: Accelerating the Inference of BERT-based Text Matching (2020.coling-main)
Copied to clipboard
| Challenge: | Recent pre-trained language models have shown state-of-the-art accuracies in text matching. |
| Approach: | They propose a BERT-based text matching model where representations and interactions are decoupled . they propose generating final matching scores using a lightweight attention network . |
| Outcome: | Experiments show that the proposed model can achieve up to 100X speed-up to BERT and RoBERTa while keeping more up to 98.7% of the performance. |
KCVR: Knowledge-Centric Video Reconstruction for Structured Pedagogical Summarization via Dynamic Graph Planning (2026.acl-long)
Copied to clipboard
Jingjiang Liu, Jia Zhu, Hanghui Guo, Weijie Shi, Yue Cui, Xiaokang Jin, Yilin Wang, Qingyu Niu, Jiawei Shen, Guoqing Ma, Yidan Liang, Shimin Di, Jiajie Xu
| Challenge: | Existing summarization methods compress content for gist browsing, but they break prerequisite logic in instructional videos. |
| Approach: | They propose a framework that decouples epistemic planning from content generation. |
| Outcome: | The proposed framework outperforms strong end-to-end baselines on Knowledge Progression Consistency and Learning Objective Coverage. |
Simple Yet Effective Synthetic Dataset Construction for Unsupervised Opinion Summarization (2023.findings-eacl)
Copied to clipboard
| Challenge: | generating aspect-specific and general opinion summaries is challenging due to the lack of annotated data. |
| Approach: | They propose two unsupervised approaches to generate aspect-specific and general opinion summaries by training on synthetic datasets constructed with aspect-related review contents. |
| Outcome: | The proposed method outperforms existing methods on space and Oposum+ and on other metrics. |
SLIM: Subtrajectory-Level Elimination for More Effective Reasoning (2025.findings-emnlp)
Copied to clipboard
Xifeng Yao, Chengyuan Ma, Dongyu Lang, Yinhao Ni, Zhiwei Xu, Huarui Xie, Zihao Chen, Guang Shen, Dandan Tu, Yi Bai, Changzheng Zhang
| Challenge: | Notable examples include OpenAI’s o1/o3/o4 series and DeepSeek-R1 . |
| Approach: | They develop a framework to identify suboptimal subtrajectories based on human-established criteria . they also use a sampling algorithm to select data whose reasoning process is free from suboptimally subtravertories to the highest degree . |
| Outcome: | The proposed method reduces the number of suboptimal subtrajectories by 25.9% during the inference process. |
JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation (2026.findings-acl)
Copied to clipboard
Zhichao Shi, Xuhui Jiang, Chengjin Xu, Cangli Yao, Shengjie Ma, Yinghan Shen, Zixuan Li, Jian Guo, Yuanzhuo Wang
| Challenge: | Current evaluation methods for large language models rely on static benchmarks . limited knowledge coverage and fixed difficulties hinder the targeted optimizations resulting in superficial evaluations of LLMs - a problem that has been addressed by JudgeAgent . |
| Approach: | They propose a knowledge-driven and dynamic evaluation framework for large language models . judgeAgent leverages LLM agents equipped with context graphs to traverse knowledge structures . |
| Outcome: | The proposed framework can achieve comprehensive evaluations and facilitate effective model iterations. |
JUREX-4E: Juridical Expert-Annotated Four-Element Knowledge Base for Legal Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have introduced legal theories into LLM workflows to improve their understanding of legal texts and reasoning accuracy. |
| Approach: | They evaluate an expert-annotated four-element knowledge base covering 155 criminal charges. |
| Outcome: | The proposed model can be used to analyze criminal charges and retrieve them in legal cases. |
Improving Contextual Query Rewrite for Conversational AI Agents through User-preference Feedback Learning (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Contextual query rewriting (CQR) is a crucial component in Conversational AI agents, leveraging contextual information from previous user-agent conversations to improve comprehension of current user intent. |
| Approach: | They propose a framework to enhance the CQR model's capability in generating user preference-aligned rewrites. |
| Outcome: | The proposed framework improves the CQR model's ability to generate user preference-aligned rewrites. |
Locate and Label: A Two-stage Identifier for Nested Named Entity Recognition (2021.acl-long)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a well-studied task in natural language processing. |
| Approach: | They propose a method that generates span proposals and labels them with categories . they use boundary information of entities and partially matched spans to locate them . |
| Outcome: | The proposed method outperforms state-of-the-art models on nested NER datasets. |
The Fall of ROME: Understanding the Collapse of LLMs in Model Editing (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have found that model editing methods can cause large language models to collapse with just a single edit. |
| Approach: | They propose a method that uses prefixed keys and adds prefixes during testing to prevent model collapse. |
| Outcome: | The proposed method prevents model collapse while maintaining effectiveness, the authors show . Rank-One Model Editing (ROME) has been found to cause model collapse with just a single edit . |
The Invisible Hand: Unveiling Provider Bias in Large Language Models for Code Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have emerged as the new recommendation engines, surpassing traditional methods in both capability and scope, particularly in code generation. |
| Approach: | They propose to use a dataset to investigate a new type of bias in Large Language Models for code generation, provider bias, to determine whether the model favors specific providers. |
| Outcome: | The proposed model favors services from Google and Amazon, but without explicit directives, and can modify input code to incorporate their preferred providers without user requests. |
SciConceptMiner: A system for large-scale scientific concept discovery (2021.acl-demo)
Copied to clipboard
| Challenge: | SciConceptMiner is a self-supervised system for the capture of scientific concepts . the system is scalable to the size of documents and the number of topics it can model . |
| Approach: | They propose a self-supervised system for the automatic capture of scientific concepts from academic publications and semi-structured data. |
| Outcome: | The proposed system achieves high accuracy (94.7%) with more than 740K scientific concepts. |
A Web-scale system for scientific knowledge exploration (P18-4)
Copied to clipboard
| Challenge: | a system that organizes scientific knowledge into a hierarchical concept structure is needed to enable efficient exploration of Web-scale knowledge. |
| Approach: | They propose a system that organizes scientific knowledge into a hierarchical concept structure . system allows researchers to identify hundreds of thousands of scientific concepts . it also allows researchers tagging scientific publications into millions of concepts based on text and graph structure based model . |
| Outcome: | The proposed system builds the most comprehensive cross-domain scientific concept ontology published to date, with more than 200 thousand concepts and over one million relationships. |
Scaling Collaborative Effort with Agents (2026.findings-acl)
Copied to clipboard
Shannon Zejiang Shen, Valerie Chen, Ken Gu, Alexis Ross, Zixian Ma, Jillian Ross, Alex Gu, Chenglei Si, Wayne Chi, Andi Peng, Jocelyn J Shen, Ameet Talwalkar, Tongshuang Wu, David Sontag
| Challenge: | Current evaluations of agents focus on producing high-quality, final outputs in one shot, failing to account for the inherently iterative nature of many real-world problems. |
| Approach: | They propose a framework that captures how an agent’s utility grows with increasing user involvement. |
| Outcome: | The proposed framework captures how an agent’s utility grows with increasing user involvement, revealing a missing ingredient in agent design: the ability to sustain engagement and scaffold user understanding. |
Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model (2024.emnlp-main)
Copied to clipboard
Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan, Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, Yueting Zhuang
| Challenge: | Using large language models, large multimodal models struggle with basic tasks like reading time from a clock and planning a route using a road map. |
| Approach: | They propose a multimodal self-instruct that synthesizes massive abstract images and visual reasoning instructions. |
| Outcome: | The proposed model synthesizes 11,193 abstract images and reasoning instructions across eight visual scenarios. |
Adversarial Self-Supervised Data-Free Distillation for Text Classification (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation algorithms rely on the accessibility of the training dataset, which may be unavailable due to privacy issues. |
| Approach: | They propose a data-free distillation method for a pre-trained transformer-based model that uses plug & play Embedding Guessing to craft pseudo embeddings from the teacher's hidden knowledge. |
| Outcome: | The proposed method is the first data-free distillation framework designed for NLP tasks. |
Surfer100: Generating Surveys From Web Resources, Wikipedia-style (2022.lrec-1)
Copied to clipboard
Irene Li, Alex Fabbri, Rina Kawamura, Yixin Liu, Xiangru Tang, Jaesung Tae, Chang Shen, Sally Ma, Tomoe Mizutani, Dragomir Radev
| Challenge: | Recent work on Wikipedia page generation focuses on generating the initial leading paragraph of a page, while recent pretrained language models improve upon both extractive and abstractive steps of previous models. |
| Approach: | They propose a pretrained language model that can be combined to generate Wikipedia-style summaries with sections using 100 reference human-collected surveys. |
| Outcome: | The proposed approach is compared with existing methods with 100 human-collected surveys. |