Papers by Leqi Shen
Beyond Logits: Aligning Feature Dynamics for Effective Knowledge Distillation (2025.acl-long)
Copied to clipboard
Guoqiang Gong, Jiaxing Wang, Jin Xu, Deping Xiang, Zicheng Zhang, Leqi Shen, Yifeng Zhang, JunhuaShu JunhuaShu, ZhaolongXing ZhaolongXing, Zhen Chen, Pengzhang Liu, Ke Zhang
| Challenge: | Knowledge distillation (KD) compresses large language models into lightweight versions called student models. |
| Approach: | They propose to align the entire feature dynamics between teacher and student models by using two additional loss terms to achieve this. |
| Outcome: | The proposed method matches the entire feature dynamics between teacher and student models rather than just the final states. |
AdaTP: Attention-Debiased Token Pruning for Video Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing visual token compression methods rely on attention scores but have inherent biases . global and local attention biased scores cause excessive computational overhead . |
| Approach: | They propose a token pruning pipeline that targets global and local attention biases . the pipeline is designed to reduce computational overhead of Video Large Language Models based on visual tokens compiled from multiple video frames . |
| Outcome: | The proposed method significantly reduces the computational overhead of Video Large Language Models while retaining the performance of vanilla models. |