Papers by Deyuan Li
Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging (2024.emnlp-main)
Copied to clipboard
Deyuan Liu, Zhanyue Qin, Hairu Wang, Zhao Yang, Zecheng Wang, Fangying Rong, Qingbin Liu, Yanchao Hao, Bo Li, Xi Chen, Cunhang Fan, Zhao Lv, Dianhui Chu, Zhiying Tu, Dianbo Sui
| Challenge: | Existing methods for parameter pruning fail to utilize the knowledge from pruned parameters. |
| Approach: | They propose a method that uses manifold learning and the Information Bottleneck measure to merge similar layers to preserve model performance. |
| Outcome: | The proposed method outperforms pruning methods on multiple datasets and LLMs with quantization and achieves substantial compression ratios. |
M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks for foundation models in understanding scientific literature focus on single-document tasks. |
| Approach: | They propose a multi-modal, multi-document scientific question answering benchmark . it uses expert-annotated questions that span 70 natural language processing paper clusters . |
| Outcome: | The proposed benchmarks underperform human experts in multi-modal reasoning and retrieval of scientific data. |