Papers by Shuhao Xing
Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law (2025.acl-long)
Copied to clipboard
Qiming Ge, Shuhao Xing, Songyang Gao, Yunhua Zhou, Yicheng Zou, Songyang Zhang, Zhi Chen, Hang Yan, Qi Zhang, Qipeng Guo, Kai Chen
| Challenge: | Large language models have demonstrated impressive performance across a wide range of tasks, but this achievement comes with the trade-off of significant computational demands. |
| Approach: | They propose a scaling law that decomposes the overall validation loss and assigns different importance weights to tokens to assess a specific meta-capability. |
| Outcome: | The proposed model can predict the loss trending of models across different levels of computation without a gap between validation loss and model's downstream capabilities. |
Code Needs Comments: Enhancing Code LLMs with Comment Augmentation (2024.findings-acl)
Copied to clipboard
Demin Song, Honglin Guo, Yunhua Zhou, Shuhao Xing, Yudong Wang, Zifan Song, Wenwei Zhang, Qipeng Guo, Hang Yan, Xipeng Qiu, Dahua Lin
| Challenge: | Large Language Models (LLMs) require a deep understanding of programming languages and their correlation with natural languages (NLs). |
| Approach: | They propose a data augmentation method that generates comments for existing code and a filtering strategy that filters out code data poorly correlated with natural language. |
| Outcome: | The proposed method outperforms the model trained on the augmented data and the model further trained on data without augmentation on two widely-used programming skill benchmarks. |
How to Set the Learning Rate for Large-Scale Pre-training? (2026.findings-acl)
Copied to clipboard
| Challenge: | Optimal configuration of the learning rate (LR) is a fundamental yet formidable challenge in large-scale pre-training. |
| Approach: | They propose a Fitting Paradigm and a Transfer Paradigme to investigate fit and transfer . they propose scalability and elucidate the reasons why module-wise parameter tuning underperforms . |
| Outcome: | The proposed model reduces the search complexity by reducing the search cost by lowering the search factor. |
CoLLiE: Collaborative Training of Large Language Models in an Efficient Way (2023.emnlp-demo)
Copied to clipboard
Kai Lv, Shuo Zhang, Tianle Gu, Shuhao Xing, Jiawei Hong, Keyu Chen, Xiaoran Liu, Yuqing Yang, Honglin Guo, Tengxiao Liu, Yu Sun, Qipeng Guo, Hang Yan, Xipeng Qiu
| Challenge: | Large language models (LLMs) are increasingly pivotal in a wide range of tasks . however, the resources required for training these models necessitate efficient solutions . |
| Approach: | They propose a library that facilitates collaborative training of large language models . they use 3D parallelism, parameter-efficient fine-tuning methods and optimizers . |
| Outcome: | The proposed library has proven superior training efficiency in comparison with prevalent solutions in pre-training and fine-tuning scenarios. |