Papers by Yukang Gan
LLaMA Pro: Progressive LLaMA with Block Expansion (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies have demonstrated that pre-trained LLMs are limited in certain domains, such as programming, mathematics, biomedical, or finance. |
| Approach: | They propose a new post-pretraining method with an expansion of Transformer blocks to tune the expanded blocks using only new corpus, efficiently and effectively improving the model’s knowledge while mitigating forgetting. |
| Outcome: | The proposed model outperforms existing models in programming and math and its instruction-following counterpart LLaMA Pro-8.3B in general tasks, programming, and mathematics. |