Papers by Huawei Shen
Steering Away from Refusal: A Black-box Jailbreak Method Based on First-Token Distribution (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to analyze black-box jailbreaks lack direct optimization signals to refine adversarial prompts. |
| Approach: | They propose a distribution-jailbreak attack method that selects effective jailbreak templates and iteratively optimizes adversarial suffixes by maximizing the KL divergence from the standard refusal distribution. |
| Outcome: | The proposed method achieves state-of-the-art Attack Success Rate (ASR) on all tested open-source models and delivers over 94% ASR on GPT-4.1. |
Transductive Learning for Unsupervised Text Style Transfer (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for style transfer are based on an inductive learning approach, which represents the style as embeddings, decoder parameters, or discriminator parameters and directly applies these general rules to the test cases. |
| Approach: | They propose a retrieval-based context-aware style representation that involves top-K relevant sentences in the target style in the transfer process. |
| Outcome: | The proposed method outperforms several strong baselines and is general and effective to the task of unsupervised style transfer. |
ReFSQL: A Retrieval-Augmentation Framework for Text-to-SQL Generation (2023.findings-emnlp)
Copied to clipboard
Kun Zhang, Xiexiong Lin, Yuanzhuo Wang, Xin Zhang, Fei Sun, Cen Jianhe, Hexiang Tan, Xuhui Jiang, Huawei Shen
| Challenge: | Existing methods that align natural language with SQL Language underestimate inherent structural characteristics of SQL and lead to structure errors. |
| Approach: | They propose a retrieval-argument framework that aligns natural language with SQL Language and trains one encoder-decoder-based model to fit all questions. |
| Outcome: | The proposed framework improves accuracy and robustness of text-to-SQL generation on five datasets. |
Learning from Mistakes: Negative Reasoning Samples Enhance Out-of-Domain Generalization (2026.acl-long)
Copied to clipboard
Tian Xueyun, MingHua Ma, Bingbing Xu, Nuoyan Lyu, Wei Li, Heng Dong, Zheng Chu, Yuanzhuo Wang, Huawei Shen
| Challenge: | Recent studies show that supervised fine-tuning (SFT) is a common approach for reasoning in large language models. |
| Approach: | They propose to use supervised fine-tuning (SFT) on chain-of-thought trajectories demonstrations . they find that incorporating negative traxories yields substantial OOD generalization gains . |
| Outcome: | The proposed scheme yields 5.51% OOD gain over positive-only training. |
Differentiated Vision: Unveiling Entity-Specific Visual Modality Requirements for Multimodal Knowledge Graph (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to extract features from images of entities overlook varying relevance of visual information across entities. |
| Approach: | a new model integrates structural and multimodal information of entities into a multimodal knowledge graph . a model evaluates the necessity of visual modality for each entity based on its attributes . |
| Outcome: | The proposed model improves on existing methods by adjusting visual data to different entity types. |
SafetyQuizzer: Timely and Dynamic Evaluation on the Safety of LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been used to evaluate the safety of their users . however, evaluation questions in current benchmarks are too straightforward and difficult to update with practical relevance due to their lack of correlation with real-world events. |
| Approach: | They propose a question-generation framework to evaluate the safety of LLMs in the Chinese context. |
| Outcome: | The proposed framework reduces decline rate while maintaining similar attack success rate. |
The Mirage of Model Editing: Revisiting Evaluation in the Wild (2025.acl-long)
Copied to clipboard
| Challenge: | despite near-perfect results, effectiveness of model editing in real-world applications remains unclear. |
| Approach: | They propose QAEdit and WILD to better reflect real-world use of model editing . they propose a benchmark aligned with widely used question answering datasets and a task-agnostic evaluation framework . |
| Outcome: | The proposed QAEdit benchmark and WILD evaluation framework show that current models perform worse than previously reported. |
MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies on knowledge editing focus on monolingual scenarios, neglecting the complexities presented by multilingual contexts and multi-hop reasoning. |
| Approach: | They propose a benchmark to evaluate the adaptability of multilingual knowledge editing methods. |
| Outcome: | The proposed benchmark evaluates the adaptability of multilingual knowledge editing methods across five languages. |
Do We Always Need Query-Level Workflows? Rethinking Agentic Workflow Generation for Multi-Agent Systems (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches generate workflows either at task level or query level, but their relative costs and benefits remain unclear. |
| Approach: | They propose a query-level workflow generation framework that generates tasks at task level and query level. |
| Outcome: | The proposed framework reduces token usage by up to 83% compared to existing approaches . it maintains competitive performance with an average degradation of just 0.61% compared with existing approaches across multiple datasets . |
ALiiCE: Evaluating Positional Fine-grained Citation Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing research on citation generation is limited to sentence-level statements . positional fine-grained citations can appear anywhere within sentences . |
| Approach: | They propose a framework that allows LLMs to generate citations from sentences . they use dependency tree-based methods to parse sentence-level claims into atomic claims . |
| Outcome: | The proposed framework evaluates citation quality using three metrics including positional fine-grained citation recall, precision, and coefficient of variation of citation positions. |
HAG: Hierarchical Demographic Tree-based Agent Generation for Topic-Adaptive Simulation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to Agent-Based Modeling fail to adapt to unseen topics absent from data. |
| Approach: | They propose a Hierarchical Agent Generation framework that formalizes population generation as a two-stage decision process. |
| Outcome: | The proposed framework outperforms baseline models in a multi-domain benchmark and comprehensive evaluation framework. |
Think Before You Speak: Cultivating Communication Skills of Large Language Models via Inner Monologue (2024.findings-naacl)
Copied to clipboard
| Challenge: | emergence of large language models (LLMs) improves capabilities of dialogue systems . but they lack communication skills, which make them more like information seeking tools . |
| Approach: | They propose to empower LLMs with communication skills through inner monologues . they use a benchmark to evaluate the dialogue generation ability of the model . |
| Outcome: | The proposed model outperforms the baselines in the evaluation of communication skills. |
PRDetect: Perturbation-Robust LLM-generated Text Detection Based on Syntax Tree (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent methods for detecting LLM-generated text have shown impressive performance, but in real-world scenarios, users often introduce perturbations to the text. |
| Approach: | They propose a method that detects syntactic trees that are minimally affected by perturbations and exhibit distinct differences between human-written and LLM-generated text. |
| Outcome: | The proposed method shows that it is significantly better against perturbations on the HC3 and GPT-3.5-mixed datasets and also has the shortest time expenditure. |
BERM: Training the Balanced and Extractable Representation for Matching to Improve Generalization Ability of Dense Retrieval (2023.acl-long)
Copied to clipboard
| Challenge: | Dense retrieval has shown promise in the first-stage retrieval process when trained on in-domain labeled datasets. |
| Approach: | They propose a method to capture matching signal to improve generalization of dense retrieval by capturing matching signal between two texts. |
| Outcome: | The proposed method can be combined with different training methods to improve generalization ability without additional inference overhead and target domain data. |
Training a Utility-based Retriever Through Shared Context Attribution for Retrieval-Augmented Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Utility-based retrieval has emerged as a promising topic for downstream tasks . however, capturing passage utility accurately remains unexplored due to insufficient understanding . |
| Approach: | They propose a framework for training utility-based retrievers in Retrieval-Augmented Language Models . it incorporates multi-task generalization and inter-passage interaction to improve performance . |
| Outcome: | The proposed framework improves performance on ten datasets across different tasks. |
ToolCoder: A Systematic Code-Empowered Tool Learning Framework for Large Language Models (2025.acl-long)
Copied to clipboard
Hanxing Ding, Shuchang Tao, Liang Pang, Zihao Wei, Jinyang Gao, Bolin Ding, Huawei Shen, Xueqi Cheng
| Challenge: | Existing approaches to tool learning rely on hand-crafted prompts and natural language reasoning, making multi-step planning difficult and lacking precise error diagnosis and reflection mechanisms. |
| Approach: | They propose a framework that reformulates tool learning as a code generation task. |
| Outcome: | The proposed framework achieves superior performance in task completion accuracy and execution reliability compared to existing approaches. |
Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that LLMs face challenges in effectively using retrieved information . authors propose a method that considers LLM as "Information Refiner" |
| Approach: | They propose a method that considers LLMs as "Information Refiners" they propose INFO-RAG, which is low-cost and general across various tasks . |
| Outcome: | The proposed method improves performance of LLaMA2 by 9.39% relative points . it is low-cost and general across various tasks, and is robust and in-context learning is possible . |
Chain-of-Memory: Lightweight Memory Construction with Dynamic Evolution for LLM Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Existing paradigms for large language model (LLM) agents use memory construction and retrieval-augmented generation. |
| Approach: | They propose a framework that advocates for a paradigm shift toward lightweight construction paired with sophisticated utilization. |
| Outcome: | Experiments show that CoM outperforms baselines with accuracy gains of 7.5%–10.4% while reducing computational overhead to approximately 2.7% of token consumption and 6.0% of latency compared to complex memory architectures. |
When to Trust LLMs: Aligning Confidence with Response Quality (2024.findings-acl)
Copied to clipboard
Shuchang Tao, Liuyi Yao, Hanxing Ding, Yuexiang Xie, Qi Cao, Fei Sun, Jinyang Gao, Huawei Shen, Bolin Ding
| Challenge: | Existing methods express reliability by confidence level, but lack objective guidance . Existing approaches express reliability but lack guidance on when to trust LLMs . |
| Approach: | They propose a reward-based approach to align confidence with quality to ensure reliability . they propose 'conqORD' to help model to verbalize greater confidence for higher quality responses . |
| Outcome: | Experiments show that CONQORD significantly improves confidence and response accuracy . the proposed approach can be used to determine reliability of large language models . |
Qsnail: A Questionnaire Dataset for Sequential Question Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Questionnaires are a professional research methodology used for qualitative and quantitative analysis of human opinions, preferences, and behaviors. |
| Approach: | They propose a questionnaire-based dataset that consists of 13,168 human-written questionnaires. |
| Outcome: | The proposed dataset contains 13,168 human-written questionnaires gathered from online platforms. |
PKAD: Pretrained Knowledge is All You Need to Detect and Mitigate Textual Backdoor Attacks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current defense methods can be classified into inference-time and training-time ones based on their execution phase. |
| Approach: | They propose a two-stage poison detection strategy using pre-trained language models to detect poisoned samples before model training. |
| Outcome: | The proposed method achieves better performance than current methods more quickly and with fewer training costs. |
Meta-CQG: A Meta-Learning Framework for Complex Question Generation over Knowledge Bases (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods train one encoder-decoder-based model to fit all questions . however, such a one-size-fits-all strategy may not perform well for complex questions involving multiple KB relations or functional constraints. |
| Approach: | They propose a meta-learning framework for complex question generation over knowledge bases . they propose he meta-trained generator can acquire universal meta-knowledge . |
| Outcome: | The proposed framework can acquire universal and transferable meta-knowledge and quickly adapt to long-tailed samples under different dimensions. |
Jailbreak LLMs through Internal Stance Manipulation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to exploit LLMs' inherent safety mechanism, including GCG and AutoDAN, are ineffective for certain malicious requests. |
| Approach: | They propose a method that generates jailbreak prompts to suppress a refusal stance and induce affirmative responses by modifying adversarial prompts. |
| Outcome: | The proposed method outperforms the best baseline approach in Llama-2-7b-chat and achieves a 92.2% success rate across all models. |
Enhancing Training Data Attribution for Large Language Models with Fitting Error Consideration (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are difficult to interpret due to their black-box nature and randomness. |
| Approach: | They propose a new method which enhances influence functions by addressing fitting errors by eliminating knowledge bias present in the base model before fine-tuning. |
| Outcome: | The proposed method outperforms existing methods and achieves an average AUC of 91.64%. |
Do LLMs Play Dice? Exploring Probability Distribution Sampling in Large Language Models for Behavioral Simulation (2025.coling-main)
Copied to clipboard
| Challenge: | LLMs are used to emulate sequential decision-making processes of humans . however, their ability to perform probabilistic sampling is limited . |
| Approach: | They propose to use large language models (LLMs) as agents to emulate the sequential decision-making processes of humans represented as Markov decision-makers (MDPs). |
| Outcome: | The proposed models can understand probabilities, but struggle with sampling precision . integrating coding tools can improve sampling precision, but this level of sampling precision still makes it difficult to simulate human behavior as agents. |
Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs (2025.emnlp-main)
Copied to clipboard
Hexiang Tan, Fei Sun, Sha Liu, Du Su, Qi Cao, Xin Chen, Jingang Wang, Xunliang Cai, Yuanzhuo Wang, Huawei Shen, Xueqi Cheng
| Challenge: | Existing detection methods fail to account for **self-consistent error** . study identifies self-consistency errors and evaluates them . |
| Approach: | They propose a method that fuses hidden state evidence from an external verifier LLM to detect self-consistent errors. |
| Outcome: | The proposed method significantly enhances performance on self-consistent errors across three LLM families. |
LLMDet: A Third Party Large Language Models Generated Text Detection Tool (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing detection tools rely on access to LLMs and can only distinguish between machine-generated and human-authored text. |
| Approach: | They propose a model-specific, secure, efficient, and extendable detection tool that can source text from specific LLMs. |
| Outcome: | The proposed tool can source text from specific LLMs, such as GPT-2, OPT, LLaMA, and others. |
Unlocking the Power of Large Language Models for Entity Alignment (2024.acl-long)
Copied to clipboard
Xuhui Jiang, Yinghan Shen, Zhichao Shi, Chengjin Xu, Wei Li, Zixuan Li, Jian Guo, Huawei Shen, Yuanzhuo Wang
| Challenge: | Entity Alignment (EA) is a crucial step in unifying data from heterogeneous sources and plays a critical role in data-driven AI applications. |
| Approach: | They propose a framework that incorporates large language models to improve EA. |
| Outcome: | The proposed framework incorporates large language models (LLMs) to improve EA accuracy while preserving efficiency. |
MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Space (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to multi-aspect controllable text generation require expensive iteration / searching within the discrete text space during the decoding stage, resulting in a degradation of text quality due to the domain discrepancies between different aspects. |
| Approach: | They propose a framework that estimates compact latent space for multiple aspects and performs efficient Sampling with a fast sampler to eliminate domain discrepancies. |
| Outcome: | The proposed framework outperforms baselines on attribute relevance and textual quality while maintaining a high inference speed. |
SimOAP: Improve Coherence and Consistency in Persona-based Dialogue Generation via Over-sampling and Post-evaluation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing work on large-scale corpora-based language models is limited and hard to generalize to all types of pre-trained language models. |
| Approach: | They propose a two-stage SimOAP strategy that over-samples and post-evaluates large-scale responses from existing models and selects a good response based on multiple evaluation metrics. |
| Outcome: | The proposed strategy outperforms baseline and automatic evaluation strategies in both automatic and human evaluations. |
Projecting Out the Malice: A Global Subspace Approach to LLM Detoxification (2026.acl-long)
Copied to clipboard
Zenghao Duan, Zhiyi Yin, Zhichao Shi, Liang Pang, Shaoling Jing, Zihe Huang, Jiayi Wu, Yu Yan, Jingcheng Deng, Huawei Shen, Xueqi Cheng
| Challenge: | Large language models (LLMs) exhibit exceptional performance but pose inherent risks of generating toxic content. |
| Approach: | They propose a method that removes toxic subspaces from FFN parameters . they propose to use a lightweight method to eliminate toxic subespaces . |
| Outcome: | The proposed method achieves SOTA detoxification while preserving general capabilities without large-scale retraining. |
Related Knowledge Perturbation Matters: Rethinking Multiple Pieces of Knowledge Editing in Same-Subject (2025.naacl-short)
Copied to clipboard
Zenghao Duan, Wenbin Duan, Zhiyi Yin, Yinghan Shen, Shaoling Jing, Jie Zhang, Huawei Shen, Xueqi Cheng
| Challenge: | Existing knowledge editing methods struggle when tasked with editing multiple related knowledge pieces for the same subject. |
| Approach: | They propose a benchmark to assess the effectiveness of knowledge editing methods . they use same-subject edits to ensure comprehensive updates to entity-centric knowledge . |
| Outcome: | The proposed method over-relys on subject information, neglecting other critical factors, resulting in reduced editing effectiveness. |
Low-Entropy Watermark Detection via Bayes’ Rule Derived Detector (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for text watermarking ignore strong evidences embedded in low-entropy tokens, causing statistical measures to falsely indicate the absence of a watermark. |
| Approach: | They propose a Bayes' Rule derived watermark Detector which exploits watermark information from every token by leveraging the posterior probability of watermark’s presence. |
| Outcome: | The proposed method achieves 50% and 70% relative improvements over baselines in code generation and math problem-solving tasks. |
ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing omni-multimodal large language models lack incomplete modality support or lack autonomous proactive monitoring. |
| Approach: | They propose a real-time omni-multimodal assistant for unified reactive and proactive interaction that decouples response initiation from generation to ensure precise triggering without task conflict. |
| Outcome: | The proposed model achieves state-of-the-art performance on proactive tasks while competing in reactive settings. |
The Evolution of Thought: Tracking LLM Overthinking via Reasoning Dynamics Analysis (2026.acl-long)
Copied to clipboard
Zihao Wei, Liang Pang, Jiahao Liu, Wenjie Shi, Jingcheng Deng, Shicheng Xu, Zenghao Duan, Jingang Wang, Fei Sun, Huawei Shen, Xueqi Cheng
| Challenge: | Explicit reasoning trajectories increase performance but often trigger overthinking . despite its importance, this study examines how each step of reasoning affects the final outcome . |
| Approach: | They propose a Reasoning Completion Point Detector that detects the RCP by monitoring rank dynamics of termination tokens. |
| Outcome: | The proposed method reduces token usage by up to 44% while preserving accuracy. |
Adaptive Information Seeking for Open-Domain Question Answering (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing iterative approaches to open-domain question answering use predefined strategies . e.g., BM25, DPR, and hyperlink are defined as actions . |
| Approach: | They propose a novel adaptive information-seeking strategy for open-domain question answering . they propose to use a partially observed Markov decision process to select a proper retrieval action . |
| Outcome: | Experiments on SQuAD Open and HotpotQA fullwiki show that AISO outperforms baseline methods with predefined strategies in retrieval and answer evaluations. |
Following the Autoregressive Nature of LLM Embeddings via Compression and Alignment (2025.emnlp-main)
Copied to clipboard
Jingcheng Deng, Zhongtao Jiang, Liang Pang, Zihao Wei, Liwei Chen, Kun Xu, Yang Song, Huawei Shen, Xueqi Cheng
| Challenge: | Experimental results demonstrate that our method significantly outperforms traditional contrastive learning approaches when using the same amount of data. |
| Approach: | They propose a new contrastive learning method built on embedding conditional probability distributions that integrates two tasks: information compression and conditional distribution alignment. |
| Outcome: | The proposed method outperforms traditional contrastive learning approaches and achieves comparable performance to state-of-the-art models when using the same amount of data. |
Plot Retrieval as an Assessment of Abstract Semantic Association (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing information retrieval datasets cannot capture abstract semantic associations well. |
| Approach: | They propose a task that retrieves relevant plots from the book for a query using a labeled dataset. |
| Outcome: | The proposed task can be used to evaluate the performance of IR models on the novel task Plot Retrieval. |
The Fall of ROME: Understanding the Collapse of LLMs in Model Editing (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have found that model editing methods can cause large language models to collapse with just a single edit. |
| Approach: | They propose a method that uses prefixed keys and adds prefixes during testing to prevent model collapse. |
| Outcome: | The proposed method prevents model collapse while maintaining effectiveness, the authors show . Rank-One Model Editing (ROME) has been found to cause model collapse with just a single edit . |
RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on retrieval-augmented language models has two main problems: determining what information to retrieve and effectively combining retrieved information during generation. |
| Approach: | They propose a retrieval-augmented language model that captures current and future information from source and target text into a latent space. |
| Outcome: | The proposed model is more efficient than explicit raw text, but limited by context length and noise. |
From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to align large language models with human preferences suffer from inconsistent scoring and suboptimal alignment. |
| Approach: | They propose a dual-consistency framework that aligns partial sequences with human preferences. |
| Outcome: | The proposed framework significantly reduces granularity discrepancies and improves GPT-4 evaluation scores. |