Papers by Shengyu Mao
VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on pushing weight-only quantization to extremely low-bit due to numerical representation limitations. |
| Approach: | They propose a vector-based quantization approach that pushes LLMs to extremely low-bit . they propose scalar-based weight quantization that reduces memory requirements and optimizes storage costs . |
| Outcome: | The proposed method reduces model quantization perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on mistral-7B, 4.41-7.34, on llaMA-3 on QA tasks on average. |
Knowledge Rumination for Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that pre-trained language models lack the capacity to handle knowledge-intensive tasks alone. |
| Approach: | They propose a new paradigm to help pre-trained language models utilize latent knowledge without retrieving it from external corpus. |
| Outcome: | The proposed paradigm can be applied to pre-trained language models without retrieving external knowledge from the corpus. |
SPEECH: Structured Prediction with Energy-Based Event-Centric Hyperspheres (2023.acl-long)
Copied to clipboard
| Challenge: | Experiments on two unified-annotated event datasets indicate that SPEECH is predominant in event detection and event-relation extraction tasks. |
| Approach: | They propose to model complex dependency among event structured components with energy-based energy-modeling and represent event classes with simple but effective hyperspheres. |
| Outcome: | Experiments on two unified-annotated event datasets show that SPEECH is predominant in event detection and event-relation extraction tasks. |
Editing Conceptual Knowledge for Large Language Models (2024.findings-emnlp)
Copied to clipboard
Xiaohan Wang, Shengyu Mao, Shumin Deng, Yunzhi Yao, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen, Ningyu Zhang
| Challenge: | Existing knowledge editing methods can modify concept-level definitions, but they can distort instantial knowledge in LLMs, leading to poor performance. |
| Approach: | They construct a benchmark dataset ConceptEdit and establish new metrics for evaluation to investigate the editing capability of LLMs. |
| Outcome: | The proposed methods can modify concept definitions but can distort instantial knowledge in LLMs, leading to poor performance. |
Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atoms (2025.acl-long)
Copied to clipboard
| Challenge: | Recent research has explored the use of sparse autoencoders (SAE) to disentangle knowledge in high-dimensional spaces for steering. |
| Approach: | They propose a method that isolates and manipulates disentangled knowledge components to enhance safety by using sparse autoencoders to disentangle knowledge in high-dimensional spaces for steering. |
| Outcome: | The proposed method is able to isolate and manipulate disentangled knowledge components to enhance safety in large reasoning models. |
EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models (2024.acl-demos)
Copied to clipboard
Peng Wang, Ningyu Zhang, Bozhong Tian, Zekun Xi, Yunzhi Yao, Ziwen Xu, Mengru Wang, Shengyu Mao, Xiaohan Wang, Siyuan Cheng, Kangwei Liu, Yuansheng Ni, Guozhou Zheng, Huajun Chen
| Challenge: | Large Language Models (LLMs) suffer from knowledge cutoff or fallacy issues, which means they are unaware of unseen events or generate text with incorrect facts owing to outdated/noisy data. |
| Approach: | They propose an easy-to-use knowledge editing framework for Large Language Models that allows users to easily edit updated knowledge and adjust undesired behavior while minimizing the impact on unrelated inputs. |
| Outcome: | The proposed framework surpasses traditional fine-tuning in terms of reliability and generalization. |
Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on large-scale labeled support sets are not feasible in practical scenarios. |
| Approach: | They introduce a language model-based determinant point process that considers uncertainty and diversity of unlabeled instances for optimal selection. |
| Outcome: | The proposed method can effectively select canonical examples on 9 NLU and 2 Generation datasets. |
RaFe: Ranking Feedback Improves Query Rewriting for RAG (2024.findings-emnlp)
Copied to clipboard
Shengyu Mao, Yong Jiang, Boli Chen, Xiao Li, Peng Wang, Xinyu Wang, Pengjun Xie, Fei Huang, Huajun Chen, Ningyu Zhang
| Challenge: | Large Language Models (LLMs) and Retrieval Augmentation Generation (RAG) techniques have evolved to enhance document retrieval by reformulating queries. |
| Approach: | They propose a framework for training query rewriting models that leverages a reranker framework. |
| Outcome: | The proposed framework provides ranking feedback aligned well with the rewriting objectives without needing signals from annotations and supports both online and offline training models. |