Papers by Yifan Peng
AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories (2024.findings-emnlp)
Copied to clipboard
Yifan Song, Weimin Xiong, Xiutian Zhao, Dawei Zhu, Wenhao Wu, Ke Wang, Cheng Li, Wei Peng, Sujian Li
| Challenge: | Existing studies focus on specialized agents designed for particular tasks. |
| Approach: | They propose to scale annotated interaction trajectories and fine-tune LLMs on AgentBank to get a series of agent models, Samoyed. |
| Outcome: | The proposed model can scale to get generalized agent capabilities. |
VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning (2025.naacl-long)
Copied to clipboard
Yifan Peng, Krishna C Puvvada, Zhehuai Chen, Piotr Zelasko, He Huang, Kunal Dhawan, Ke Hu, Shinji Watanabe, Jagadeesh Balam, Boris Ginsburg
| Challenge: | Recent studies have augmented large language models (LLMs) with speech capabilities, leading to the development of speech language models. |
| Approach: | They propose a single-stage joint speech-text SFT approach for training SpeechLMs . their model combines text-only SFT data with three types of speech-related data . |
| Outcome: | The proposed model outperforms previous SpeechLMs on speech-based QA tasks while maintaining original speech-only capabilities. |
UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions (2024.naacl-long)
Copied to clipboard
Siddhant Arora, Hayato Futami, Jee-weon Jung, Yifan Peng, Roshan Sharma, Yosuke Kashiwagi, Emiru Tsunoo, Karen Livescu, Shinji Watanabe
| Challenge: | Recent studies leverage large language models with multi-tasking capabilities, using natural language prompts to guide the model’s behavior and surpassing performance of task-specific models. |
| Approach: | They adapt a pre-trained automatic speech recognition model to additional tasks using single-token task specifiers. |
| Outcome: | The proposed model can generalize to new datasets and languages for seen task types. |
Improving Retrieval-Augmented Generation without Taxonomy-based Error Categorization (2026.acl-short)
Copied to clipboard
| Challenge: | Recent work implicitly assumes reliable critic feedback and focuses on planning strategies, while paying limited attention to the robustness of the error correction process itself. |
| Approach: | They propose a response-action learning paradigm that maps flawed RAG outputs to error-mitigating action plans without explicit criticism. |
| Outcome: | The proposed model improves the factual accuracy of large language model outputs without explicit error categorization. |
PACE: Prefix-Protected and Difficulty-Aware Compression for Efficient Reasoning (2026.findings-acl)
Copied to clipboard
Ruixiang Feng, Yuntao Wen, Silin Zhou, Ke Shi, Yifan Wang, Ran Le, Zhenwei An, Zongchao Chen, Chen Yang, Guangyue Peng, Yiming Jia, Dongsheng Wang, Tao Zhang, Lisi Chen, Yang Song, Shen Gao, Shuo Shang
| Challenge: | Existing LRMs often suffer from "overthinking" and excessively long reasoning traces . a dual-level framework for length compression of LRM is proposed . |
| Approach: | They propose a framework for prefix-protected and difficulty-aware compression under hierarchical supervision. |
| Outcome: | The proposed framework reduces token usage while improving accuracy on math benchmarks. |
RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging (2025.emnlp-main)
Copied to clipboard
Bowen Wang, Haiyuan Wan, Liwen Shi, Chen Yang, Peng He, Yue Ma, Haochen Han, Wenhao Li, Tiao Tan, Yongjian Li, Fangming Liu, Gong Yifan, Sheng Zhang
| Challenge: | Existing models that require task labels or performance trade-offs are susceptible to catastrophic forgetting. |
| Approach: | They propose a representation-aware model merging framework for continual learning without access to historical data. |
| Outcome: | The proposed framework outperforms baselines in knowledge retention and generalization across five NLP tasks and multiple continual learning scenarios. |
ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit (2023.acl-demo)
Copied to clipboard
Brian Yan, Jiatong Shi, Yun Tang, Hirofumi Inaguma, Yifan Peng, Siddharth Dalmia, Peter Polák, Patrick Fernandes, Dan Berrebbi, Tomoki Hayashi, Xiaohui Zhang, Zhaoheng Ni, Moto Hira, Soumi Maiti, Juan Pino, Shinji Watanabe
| Challenge: | ESPnet-ST-v2 is a revamp of the open-source spoken language translation toolkit . it supports offline speech-to-text translation (ST), simultaneous speech- to-text (SST), and offline speech to-speech (S2ST) |
| Approach: | They propose to revamp the open-source ESPnet-ST toolkit to support offline speech-to-text translation, simultaneous speech- to-text and offline speech to-speech translation. |
| Outcome: | The updated version of ESPnet-ST supports offline speech-to-text translation (ST), simultaneous speech- to-text (SST), and offline speech to-speech translation (S2ST). |
RelEdit: Evaluating Conceptual Knowledge Editing in Language Models via Relational Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge editing methods struggle to reason about related conceptual knowledge effectively, despite a lack of model-level relational reasoning. |
| Approach: | They propose a benchmark to assess concept-level and instance-level relational reasoning abilities of edited models. |
| Outcome: | The proposed model obtains the best scores on the memory-based in-context editing baseline, MICE, suggesting a promising direction for model editing. |
Budget-Aware Routing for Long Clinical Text (2026.findings-acl)
Copied to clipboard
| Challenge: | Long-context capability is now a headline feature of large language models . clinical inputs are long because they are templated, redundant, and stitched from multiple sources. |
| Approach: | They propose a token-constrained subset selection problem with two design choices . they propose heuristics that balance relevance, coverage, diversity and a monotone submodular objective . |
| Outcome: | The proposed model is based on a subset selection problem with two design choices . positional heuristics perform best at low budgets in extractive tasks, while diversity-aware methods improve LLM generation. |
MARCH: Multi-Agent Radiology Clinical Hierarchy for CT Report Generation (2026.acl-short)
Copied to clipboard
| Challenge: | Automated 3D radiology report generation suffers from clinical hallucinations and lacks the iterative verification characteristic of clinical workflows. |
| Approach: | They propose a multi-agent framework that emulates the professional hierarchy of radiology departments and assigns specialized roles to distinct agents. |
| Outcome: | The proposed framework outperforms state-of-the-art models in clinical fidelity and linguistic accuracy on the RadGenome-ChestCT dataset. |
CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalization (2026.acl-long)
Copied to clipboard
Zhongyuan Peng, Yifan Yao, Kaijing Ma, Shuyue Guo, Yizhe Li, Yichi Zhang, Chenchen Zhang, Yifan Zhang, Zhouliang Yu, Luming Li, Minghao Liu, Yihang Xia, Jiawei Shen, Yuchen Wu, Yixin Cao, Zhaoxiang Zhang, Wenhao Huang, Jiaheng Liu, Ge Zhang
| Challenge: | Existing approaches to formalizing mathematical statements face limitations in accuracy, especially in the context of complex, highlevel problems that involve sophisticated mathematical reasoning. |
| Approach: | They propose a CriticLean framework that elevates the role of the critic from a passive validator to an active learning component and introduce a benchmark to measure models’ ability to distinguish semantically correct from incorrect formalizations. |
| Outcome: | The proposed framework outperforms open- and closed-source benchmarks and shows that it significantly outperformed existing models. |
A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends (2026.findings-acl)
Copied to clipboard
Yihao Ding, Siwen Luo, Yue Dai, Yanbei Jiang, Zechuan Li, Qiang Sun, Geoffrey Martin, Wei Liu, Yifan Peng
| Challenge: | Visually Rich Document Understanding (VRDU) frameworks are a key area of research . early approaches to VRDU relied on manually crafted rules and domain-specific heuristics . conventional deep learning approaches do not integrate the diverse modalities in documents . |
| Approach: | They review recent advances in MLLM-based Visually Rich Document Understanding (VRDU) their findings highlight emerging trends and promising research directions . |
| Outcome: | The proposed frameworks are scalable, reliable, and adaptable, the authors argue . their findings highlight emerging trends and promising research directions . |
Watch Every Step! LLM Agent Learning via Iterative Step-level Process Refinement (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to enhance agent performance focus on outcome rewards, which may lead to errors or suboptimal actions due to the absence of process supervision signals. |
| Approach: | They propose a step-level framework that provides detailed step-by-step guidance to enhance agent training by using Monte Carlo methods. |
| Outcome: | The proposed framework outperforms strong baselines on three tasks and shows that it is effective in augmenting efficiency and its applicability to diverse models. |
ChatKBQA: A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned Large Language Models (2024.findings-acl)
Copied to clipboard
Haoran Luo, Haihong E, Zichen Tang, Shiyao Peng, Yikai Guo, Wentai Zhang, Chenghao Ma, Guanting Dong, Meina Song, Wei Lin, Yifan Zhu, Anh Tuan Luu
| Challenge: | Existing KBQA methods address inefficient knowledge retrieval and semantic parsing errors. |
| Approach: | They propose a generatethen-retrieve KBQA framework that generates logical form and replaces entities and relations with an unsupervised retrieval method to improve both generation and retrieval more directly. |
| Outcome: | Experimental results show that ChatKBQA achieves new state-of-the-art performance on standard KBQA datasets, WebQSP, and CWQ. |
ReCUT: Balancing Reasoning Length and Accuracy in LLMs via Stepwise Trails and Preference Optimization (2025.findings-emnlp)
Copied to clipboard
Zhensheng Jin, Xinze Li, Yifan Ji, Chunyi Peng, Zhenghao Liu, Qi Shi, Yukun Yan, Shuo Wang, Furong Peng, Ge Yu
| Challenge: | Existing methods to train LLMs suffer from overthinking, leading to lengthy reasoning traces . Existing approaches to train large language models suffer from this problem . |
| Approach: | They propose a method to combine multiple reasoning chains for training LLMs . they use stepwise exploration and long-short switched sampling to evaluate reasoning paths . |
| Outcome: | The proposed method reduces reasoning lengths by approximately 30-50% . it also maintains or improves reasoning accuracy compared to baselines . |
Towards Robust Speech Representation Learning for Thousands of Languages (2024.emnlp-main)
Copied to clipboard
William Chen, Wangyou Zhang, Yifan Peng, Xinjian Li, Jinchuan Tian, Jiatong Shi, Xuankai Chang, Soumi Maiti, Karen Livescu, Shinji Watanabe
| Challenge: | XEUS is a cross-lingual encoder for universal speech that can be trained on 1 million hours of data across 4057 languages. |
| Approach: | They propose a Cross-lingual Encoder for Universal Speech that can be trained on 1 million hours of data across 4057 languages and a newly created corpus of 7400+ hours from 4057 . |
| Outcome: | The proposed model outperforms state-of-the-art models on several benchmarks and outperfies MMS 1B and w2v-BERT 2.0 v2 by 0.8% and 4.4% respectively. |
Evaluating the Expressive Appropriateness of Speech in Rich Contexts (2026.acl-long)
Copied to clipboard
Tianrui Wang, Ziyang Ma, Yizhou Peng, Haoyu Wang, Zhikang Niu, Zikang Huang, Yihao Wu, Yi-Wen Chao, Yu Jiang, Yuheng Lu, Guanrou Yang, Xuanchen Li, Hexin Liu, Chunyu Qiang, Cheng Gong, Yifan Yang, Tianchi Liu, Junyu Wang, Nana Hou, Meng Ge, Fuming You, Yang Wei, Zhongqian Sun, Hu Haifeng, Xiaobao Wang, Eng Siong Chng, Xie Chen, Longbiao Wang, Jianwu Dang
| Challenge: | Existing methods for evaluating expressive speech focus on word accuracy, naturalness, signal quality, or emotional intensity at the utterance level. |
| Approach: | They propose a framework for Evaluating Expressive Appropriateness in speech that assesses whether a speech sample aligns with the underlying communicative intent implied by its discourse-level narrative context. |
| Outcome: | The proposed framework outperforms existing speech evaluation and analysis systems on a human-annotated test set. |
Leveraging Deep Representations of Radiology Reports in Survival Analysis for Predicting Heart Failure Patient Mortality (2021.naacl-main)
Copied to clipboard
| Challenge: | Current models for survival analysis are limited in scope and require a large amount of data and expert annotations for training. |
| Approach: | They propose to use BERT-based hidden layer representations of clinical texts as covariates for proportional hazards models to predict patient survival outcomes. |
| Outcome: | The proposed method outperforms the baseline model by 5.7% across C-index and time-dependent AUC. |
OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification (2024.acl-long)
Copied to clipboard
| Challenge: | Autoregressive models can be slower during inference and have potential risks of hallucination. |
| Approach: | They propose an encoder-only speech foundation model based on Connectionist Temporal Classification. |
| Outcome: | The proposed model improves on 180k hours of public audio data for multilingual speech recognition, speech translation, and language identification. |
Less Likely Brainstorming: Using Language Models to Generate Alternative Hypotheses (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to reduce cognitive errors in MRI interpretations do not work for generating less likely outputs. |
| Approach: | They propose a task that asks a model to generate outputs that humans think are relevant but less likely to happen. |
| Outcome: | The proposed method compares with several state-of-the-art controlled text generation models via automatic and human evaluations and shows that it reduces cognitive errors in interpreting MRI findings. |
Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injection (2026.acl-long)
Copied to clipboard
Yuwei Zhang, Wenhao Yu, Shangbin Feng, Yifan Zhu, Letian Peng, Jayanth Srinivasa, Gaowen Liu, Jingbo Shang
| Challenge: | Existing knowledge injection benchmarks for large language models lack standardized testing grounds. |
| Approach: | They propose a knowledge injection benchmark that leverages recently-added and expert-curated facts from Wikipedia’s “Did You Know...” entries. |
| Outcome: | The proposed framework improves reliability accuracy by 29.1%. |
Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agent (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized open-domain dialogue agents but face challenges in multi-character role-playing (MCRP) scenarios. |
| Approach: | They propose a framework for efficient multi-character role-playing that employs a dynamic low-rank adapter strategy and distinct LoRA blocks for each character. |
| Outcome: | Neeko employs a dynamic low-rank adapter (LoRA) strategy, enabling it to adapt seamlessly to diverse characters. |
Natural Language Processing in Support of Evidence-based Medicine: A Scoping Review (2025.findings-acl)
Copied to clipboard
| Challenge: | Evidence-based medicine (EBM) is at the forefront of modern healthcare, emphasizing the use of the best available scientific evidence to guide clinical decisions. |
| Approach: | They propose to investigate the use of Natural Language Processing (NLP) techniques to identify, appraise, synthesize, apply, and disseminate evidence in EBM. |
| Outcome: | The proposed methods support the five fundamental steps of EBM—Ask, Acquire, Appraise, Apply, and Assess. |