Papers by Weixiang Xu
RQT: Hierarchical Residual Quantization for Multi-Model Compression (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for decomposing fine-tuned LLMs are sensitive to the magnitude of delta values. |
| Approach: | They propose a hierarchical quantization framework that shares low-bit integer weights across similar models. |
| Outcome: | The proposed framework achieves an average accuracy degradation of approximately 3% on fine-tuned models across mathematics, coding, chatbot, and Chinese LLMs. |
Q-Mamba: Towards more efficient Mamba models via post-training quantization (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that Mamba architectures have room for further optimization in linear projections and state caches. |
| Approach: | They propose a decoupled scale quantization scheme to mitigate outliers in states and channels by applying separate quantization scales. |
| Outcome: | The proposed method reduces memory consumption by 50% across various quantization settings, model sizes, and generation and zero-shot tasks. |
SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models (2024.acl-long)
Copied to clipboard
Weixiang Zhao, Shilong Wang, Yulin Hu, Yanyan Zhao, Bing Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, Wanxiang Che
| Challenge: | Existing methods to address catastrophic forgetting and knowledge transfer in large language models (LLMs) ignore potential of aligning the two modules to effectively address catastrophic forgetting and knowledge transfers simultaneously. |
| Approach: | They propose a Shared Attentive Learning & Selection module to align the PET learning and selection modules to address catastrophic forgetting and knowledge transfer simultaneously. |
| Outcome: | Experiments on two CL benchmarks show that the proposed framework is superior when scaled to different model sizes, different model architectures and unseen tasks. |