Papers by Xiaohui Song
BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation (2025.coling-main)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a method for reducing model size while preserving performance. |
| Approach: | They propose a method to distill large language models at the logit level by transferring knowledge from a large teacher model to a smaller student model. |
| Outcome: | The proposed method outperforms supervised fine-tuning, vanilla KL loss and five other distillation methods on 13 datasets. |
Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations (2026.acl-long)
Copied to clipboard
Wentao Hu, Yanbo Zhai, Xiaohui Hu, Mingkuan Zhao, Shanhong yu, Xue Liu, Kaidong Yu, Shuangyong Song, Xuelong Li
| Challenge: | Sparse Mixture-of-Experts models are vulnerable to hallucinations, authors say . static Top-k routing leaves "specialist experts" under-prioritized for specific tokens . |
| Approach: | They propose a training-free inference framework to awaken dormant experts . they propose 'counterfactual routing' to shift computational resources from syntax-dominant to knowledge-intensive layers . |
| Outcome: | Experiments show that CoR improves factual accuracy by 3.1% without increasing the inference budget. |
Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated exceptional performance across diverse natural language processing tasks. |
| Approach: | They propose a method for converting multi-head attention into grouped-query attention with any compression ratio of KV heads. |
| Outcome: | The proposed method can compress up to 87.5% KV heads of LLaMA2-7B model and 75% Kv heads of Sheared-LLa MA-1.3B with acceptable performance degradation. |
Supervised Prototypical Contrastive Learning for Emotion Recognition in Conversation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to capture emotions in conversation (ERC) lack the correlation between emotions and semantics, resulting in many challenges. |
| Approach: | They propose a Supervised Prototypical Contrastive Learning (SPCL) loss for the ERC task . they use a Prototype Network to leverage the supervised contrastive learning approach . |
| Outcome: | The proposed approach outperforms CoG-BART's proposed approach on three widely used benchmarks and shows that it is effective on multiple scenarios. |