Papers by Zhuoyang Song
DimA: A Parameter-efficient Fine-tuning Method with Knowledge Transfer Based on Transformer (2024.lrec-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have demonstrated impressive performance across various downstream tasks, but fine-tuning is computationally expensive and storage-intensive. |
| Approach: | They propose a parameter-efficient method called DimA which enhances the transformer architecture by increasing the dimensionality. |
| Outcome: | The proposed method achieves state-of-the-art results in GLUE and XSUM tasks while utilizing less than 1% of the original model’s parameters. |
Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Training (2024.acl-long)
Copied to clipboard
Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, LiuYiBo LiuYiBo, Qianguosun Qianguosun, Yuxin Liang, Hao Wang, Enming Zhang, Jiaxing Zhang
| Challenge: | Large language models suffer from severe hallucinations, compromising performance in knowledge-oriented QA, dialogue, and writing. |
| Approach: | They propose to enhance the information searching and reflection ability of large language models by training them in position-agnostic multi-step QA tasks to improve their model's accuracy. |
| Outcome: | The proposed model improves in multi-doc QA and other benchmarks by 13.7% absolute gain in shuffled settings, by 21.5% in passage retrieval task. |