Papers by Pengjie Ren
Retrieval Models Aren’t Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) suffer from inherent inabilities to interact with the physical world and access vast, up-to-date knowledge. |
| Approach: | They propose a tool retrieval benchmark for large language models (LLMs) that includes 7.6k diverse retrieval tasks and a corpus of 43k tools. |
| Outcome: | The proposed model performs poorly on the heterogeneous tool retrieval benchmark, resulting in low pass rate and low retrieval quality. |
Self-Supervised Position Debiasing for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for debiasing large language models require external bias knowledge or annotated non-biased samples, which is lacking for position debiases. |
| Approach: | They propose a self-supervised position debiasing framework that leverages unsupervised responses from pre-trained LLMs for debiazing without external bias knowledge. |
| Outcome: | The proposed framework outperforms existing methods in mitigating three types of position biases on eight datasets and five tasks. |
Bridging the Capability Gap: Joint Alignment Tuning for Harmonizing LLM-based Multi-Agent Systems (2025.findings-emnlp)
Copied to clipboard
Minghang Zhu, Zhengliang Shi, Zhiwei Xu, Shiguang Wu, Lingjie Wang, Pengjie Ren, Zhaochun Ren, Zhumin Chen
| Challenge: | Existing methods for fine-tuning agents are often inadequate . a multi-agent system can solve complex tasks by dividing responsibilities among specialized agents . |
| Approach: | a new framework is proposed to improve agents collaboration through iterative alignment. |
| Outcome: | The proposed framework outperforms state-of-the-art benchmarks on held-in and held-out tasks. |
SLARD: A Chinese Superior Legal Article Retrieval Dataset (2025.coling-main)
Copied to clipboard
| Challenge: | Existing retrieval methods struggle to achieve ideal results, a study finds . existing large language models lack prior knowledge of the content of superior legal articles . |
| Approach: | They propose to use a Chinese superior legal article retrieval dataset to find relevant articles with higher legal effectiveness. |
| Outcome: | The proposed dataset shows that existing retrieval methods struggle to achieve ideal results. |
Generate-then-Ground in Retrieval-Augmented Generation for Multi-hop Question Answering (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to solve multi-hop question are constrained by the retriever and the noise in the retrieved documents. |
| Approach: | They propose a framework that integrates parametric knowledge of large language models with external documents to solve a multi-hop question. |
| Outcome: | The proposed framework is based on the parametric knowledge of LLMs and external documents to solve a multi-hop question. |
Syllogistic Reasoning for Legal Judgment Analysis (2023.emnlp-main)
Copied to clipboard
Wentao Deng, Jiahuan Pei, Keyi Kong, Zhe Chen, Furu Wei, Yujun Li, Zhaochun Ren, Zhumin Chen, Pengjie Ren
| Challenge: | Legal judgment assistants are developing fast due to impressive progress of large language models. |
| Approach: | They construct and manually correct a syllogistic reasoning dataset for legal judgment analysis using large language models as benchmarks. |
| Outcome: | The proposed dataset contains 11,239 criminal cases covering 4 criminal elements, 80 charges and 124 articles. |
Intent-calibrated Self-training for Answer Selection in Open-domain Dialogues (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing answer selection models require large amounts of labeled data to produce accurate answers. |
| Approach: | They propose intent-calibrated self-training to calibrate answer labels using labeled data . they propose intentcalibration to improve quality of pseudo answer labels . |
| Outcome: | The proposed intent-calibrated answer selection paradigm outperforms baselines with 1%, 5%, and 10% labeled data on two benchmark datasets. |
Learning to Ask Conversational Questions by Optimizing Levenshtein Distance (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for estimating maximum likelihood are limited by easily learned tokens . Existing systems that generate questions based on dialogue context are limited in their ability to learn tokens. |
| Approach: | They propose a framework that optimizes the minimum Levenshtein distance through explicit editing actions. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on two benchmark datasets and generalizes well on unseen data. |
Learning to Use Tools via Cooperative and Interactive Agents (2024.findings-emnlp)
Copied to clipboard
Zhengliang Shi, Shen Gao, Xiuyi Chen, Yue Feng, Lingyong Yan, Haibo Shi, Dawei Yin, Pengjie Ren, Suzan Verberne, Zhaochun Ren
| Challenge: | Existing methods for large language models (LLMs) use one agent to iterate and execute tools, but they suffer from performance degradation when addressing practical tasks. |
| Approach: | They propose a tool learning framework that coordinates three specialized agents for tool selection, tool execution, and action calibration separately. |
| Outcome: | The proposed framework outperforms baseline models on three datasets with 14% higher success rate. |
KELE: Residual Knowledge Erasure for Enhanced Multi-hop Reasoning in Knowledge Editing (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge editing techniques show limitations when applied to multi-hop reasoning . residual single-hop knowledge causes edited models to revert to original answers . |
| Approach: | They propose a knowledge editing method that incorporates a Knowledge Erasure mechanism for Large language model Editing (KELE) they propose an erasure function for residual knowledge and an injection function for new knowledge . |
| Outcome: | The proposed method significantly improves multi-hop reasoning capability of edited models. |
UMSE: Unified Multi-scenario Summarization Evaluation (2023.findings-acl)
Copied to clipboard
| Challenge: | Summarization quality evaluation is a non-trivial task in text summarization. |
| Approach: | They propose a unified multi-scenario summarization evaluation model that shares cross-sceenario knowledge and uses a self-supervised training paradigm to optimize the model without extra human labeling. |
| Outcome: | The proposed model can achieve comparable performance with existing methods for three evaluation scenarios. |
Improving Multi-label Malevolence Detection in Dialogues through Multi-faceted Label Correlation Enhancement (2022.acl-long)
Copied to clipboard
| Challenge: | Current methods for detecting dialogue malevolence neglect label correlation. |
| Approach: | They propose to crowdsource a multi-label dataset for detecting malevolent dialogue responses and a model with label correlation enhanced CRF to measure the correlation between malevolence and negative emotions. |
| Outcome: | The proposed model outperforms the best performing baseline method on precision, recall, F1, and Jaccard score by 16.1%, 11.9%, 12.0%, and 6.1% on malevolence. |
RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue (2023.acl-long)
Copied to clipboard
| Challenge: | Evaluating open-domain dialogue systems is challenging because of the one-to-many problem. |
| Approach: | They propose a reference-based dialogue evaluation approach that leverages the pre-created utterance as reference other than the gold response to relieve the one-to-many problem. |
| Outcome: | The proposed method outperforms state-of-the-art evaluation methods on three datasets and two existing benchmarks. |
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2023.emnlp-main)
Copied to clipboard
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, Zhaochun Ren
| Challenge: | Existing work utilizes generative LLMs for Information Retrieval (IR) rather than direct passage ranking. |
| Approach: | They investigate generative LLMs such as ChatGPT and GPT-4 for relevance ranking in IR and use a test set to verify the model’s ability to rank unknown knowledge. |
| Outcome: | The proposed model outperforms a 3B supervised model on the BEIR benchmark. |
A Human-machine Collaborative Framework for Evaluating Malevolence in Dialogues (2021.acl-long)
Copied to clipboard
| Challenge: | Automatic evaluation of dialogues often shows insufficient correlation with human judgements due to the diversity of responses. |
| Approach: | They propose a framework that allows automatic evaluation of dialogues with human judgements. |
| Outcome: | The proposed framework achieves 99% evaluation accuracy with half the human effort spared. |
EmpDG: Multi-resolution Interactive Empathetic Dialogue Generation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing work on empathetic dialogue generation fails to capture the nuances of human emotion and consider the potential of user feedback. |
| Approach: | They propose a multi-resolution adversarial model - EmpDG - to generate more empathetic responses by exploiting both coarse-grained dialogue-level and fine-grounded token-level emotions. |
| Outcome: | The proposed model outperforms the state-of-the-art models in both content quality and emotion perceptivity. |
MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning (2024.acl-long)
Copied to clipboard
Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten Rijke, Zhumin Chen, Jiahuan Pei
| Challenge: | Large language models (LLMs) are the default paradigm for natural language processing (NLP) as the models’ scale and the diversity of tasks increase, fine-tuning becomes infeasible. |
| Approach: | They propose to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters and reduce their rank by 8 times . |
| Outcome: | The proposed model uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential. |
Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapse (2026.acl-long)
Copied to clipboard
Chi Zhang, Mengqi Zhang, Xiaotian Ye, Runxi Cheng, Zisheng Zhou, Ying Zhou, Pengjie Ren, Zhumin Chen
| Challenge: | Existing approaches to reducing the effects of knowledge editing are insufficiently understood. |
| Approach: | They propose a plug-and-play framework that preserves the dominant subspace of the original weights and analyzes parameter updates in the spectral basis of the weights. |
| Outcome: | The proposed framework improves editing efficacy while preserving general abilities under long-horizon sequential editing, including extreme settings with up to 20,000 edits. |
LoRE: Enhancing Search Relevance with Progressive Chain-of-Thought and Preference Alignment (2026.findings-acl)
Copied to clipboard
Chenji Lu, Zhuo Chen, Hui Zhao, Zhiyuan Zeng, Gang Zhao, Junjie Ren, null Lihaoran, Songyan Liu, Pengjie Wang, Chuan Yu, Jian Xu, Bo Zheng
| Challenge: | E-commerce search relevance is a critical component of retrieval systems. |
| Approach: | They propose a large-generative model for search relevance that trains reasoning knowledge, multi-modal understanding and rule awareness into three core competencies. |
| Outcome: | The proposed model outperforms GPT-5 in Macro-F1 and achieves 27% online gain. |
Rethinking the Alignment of Psychotherapy Dialogue Generation with Motivational Interviewing Strategies (2025.coling-main)
Copied to clipboard
| Challenge: | Motivational interviewing (MI) is a client-centered counseling technique that encourages individuals to change behaviors through emphatic conversations. |
| Approach: | They propose to use large language models to generate more controllable dialogues with explainability by prompting LLMs to predict appropriate strategies as reasoning and utilizing these strategies to guide dialogue generation. |
| Outcome: | The proposed model generates more controllable and explainable dialogues with a set of MI skills. |
Knowledge Graph Enhanced Large Language Model Editing (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for editing large language models struggle to track and incorporate changes in knowledge associated with edits, which limits the generalization ability of post-edit LLMs in processing edited knowledge. |
| Approach: | They propose a model editing method that leverages knowledge graphs to enhance LLM editing by capturing changes in associated knowledge by constructing an external graph. |
| Outcome: | The proposed method improves the generalization ability of LLMs in processing edited knowledge. |
UIPE: Enhancing LLM Unlearning by Removing Knowledge Related to Forgetting Targets (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to unlearning large language models focus on forgetting target data while overlooking the impact of logically related knowledge on the effectiveness of unlearning. |
| Approach: | They propose a method that removes knowledge highly correlated with the forgetting targets and a technique that remove logically related knowledge from the model. |
| Outcome: | The proposed method significantly improves the performance of the proposed method on the TOFU and WMDP benchmarks. |
Generative Knowledge Selection for Knowledge-Grounded Dialogues (2023.findings-eacl)
Copied to clipboard
| Challenge: | Knowledge selection is the key in knowledge-grounded dialogues (KGD), which aims to select an appropriate knowledge snippet to be used in the utterance based on dialogue history. |
| Approach: | They propose a generative approach for knowledge selection called GenKS that learns to select snippets by generating their identifiers with a sequence-to-sequence model. |
| Outcome: | The proposed approach captures intra-knowledge interaction inherently through attention mechanisms while generating their identifiers with a sequence-to-sequence model. |
Multi-Defendant Legal Judgment Prediction via Hierarchical Reasoning (2023.findings-emnlp)
Copied to clipboard
Yougang Lyu, Jitai Hao, Zihan Wang, Kai Zhao, Shen Gao, Pengjie Ren, Zhumin Chen, Fang Wang, Zhaochun Ren
| Challenge: | Existing methods for predicting judgment results for multiple defendants are ineffective. |
| Approach: | They propose a method to predict the judgment results for each defendant in multi-defendant cases . they formalize the multi-diffendant judgment process as hierarchical reasoning chains . |
| Outcome: | The proposed method can predict the judgment results for multiple defendants in multi-defendant cases. |
Generalizing Few-Shot Named Entity Recognizers to Unseen Domains with Type-Related Features (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Few-shot named entity recognition methods struggle with out-of-domain (OOD) examples due to their reliance on manual labeling for the target domain. |
| Approach: | They propose a framework to enable generalization to an unseen target domain with only a few labeled examples. |
| Outcome: | The proposed framework achieves significant performance improvements on in-domain and cross-domain datasets. |
MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter (2024.acl-long)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-tuning (PEFT) methods are limited on knowledge-intensive tasks due to the limited number of trainable parameters. |
| Approach: | They propose a mechanism that fine-tunes Large Language Models with larger adapters . they store and update the parameters of larger adapter adapters on the CPU . |
| Outcome: | The proposed method achieves comparable results to those obtained with larger memory capacities over the limited bandwidth of PCI Express (PCIe). |
Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Reality (2024.findings-acl)
Copied to clipboard
Jiahuan Pei, Irene Viola, Haochen Huang, Junxiao Wang, Moonisa Ahsan, Fanghua Ye, Jiang Yiming, Yao Sai, Di Wang, Zhumin Chen, Pengjie Ren, Pablo Cesar
| Challenge: | a fine-grained, comprehensive understanding of multimodal environments remains under-explored. |
| Approach: | They propose an automated workflow for integrating AI agents into extended reality (XR) they propose a cerebral language agent that integrates LLM with memory, planning, and interaction with XR tools and a vision-language agent . |
| Outcome: | The proposed workflow integrates AI agents seamlessly into extended reality (XR) applications for fine-grained training. |
Towards a Unified Framework for Reference Retrieval and Related Work Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for related work generation use human-annotated references as information sources. |
| Approach: | They propose a model which combines reference retrieval and related work generation processes in a unified framework based on the large language model. |
| Outcome: | The proposed model outperforms the state-of-the-art models on two wide-applied datasets. |
Answering Ambiguous Questions via Iterative Prompting (2023.acl-long)
Copied to clipboard
| Challenge: | Empirical studies show that AmbigPrompt achieves state-of-the-art or competitive results while using less memory and having a lower inference latency than competing approaches. |
| Approach: | They propose an answering model with a prompting model to address imperfections in open-domain question answering . Empirical studies show AmbigPrompt achieves state-of-the-art or competitive results . |
| Outcome: | The proposed framework improves on two commonly-used open benchmarks and achieves state-of-the-art or competitive results while using less memory and having a lower inference latency. |
KnowTuning: Knowledge-aware Fine-tuning for Large Language Models (2024.emnlp-main)
Copied to clipboard
Yougang Lyu, Lingyong Yan, Shuaiqiang Wang, Haibo Shi, Dawei Yin, Pengjie Ren, Zhumin Chen, Maarten Rijke, Zhaochun Ren
| Challenge: | Large language models (LLMs) are a default solution for many natural language processing tasks. |
| Approach: | They propose a knowledge-aware fine-tuning method to improve LLMs' knowledge awareness . they propose augmentation and comparison stages to improve accuracy and reliability . |
| Outcome: | The proposed method generates more facts with less factual error rate under fine-grained facts evaluation. |
Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025.emnlp-main)
Copied to clipboard
Yinuo Wang, Baiyang Wang, Robert Mercer, Frank Rudzicz, Sudipta Singha Roy, Pengjie Ren, Zhumin Chen, Xindi Wang
| Challenge: | achieving comprehensive trustworthiness in medical QA poses significant challenges due to complexity of healthcare data, critical nature of clinical scenarios, and multifaceted dimensions of trustworthy AI. |
| Approach: | They examine six key dimensions of trustworthiness in medical QA . they compare how each dimension is evaluated in existing LLM-based systems . |
| Outcome: | The findings show that large language models have improved patient safety and effectiveness . the models exhibit critical trust failures when deployed in clinical settings . |