| Challenge: | Existing methods for depth scaling-up rely on empirical heuristic rules for layer duplication, resulting in poor initialization and slower convergence during continual pre-training. |
| Approach: | They propose a method for learning latent parameters between layers by concatenating parameters from each layer and applying Singular Value Decomposition. |
| Outcome: | Experiments show that LESA outperforms baseline models with less than half the cost of existing methods. |
Similar Papers
Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant strides towards Artificial General Intelligence, but training them from scratch is prohibitively expensive. |
| Approach: | They propose to continuously pre-train LLMs from existing pre-trained LLM models by using a set of parameters instead of randomly initializing them. |
| Outcome: | The proposed approach saves significant resources and accelerates convergence and performance. |
LLMaAA: Making Large Language Models as Active Annotators (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing supervised learning methods in natural language processing require large amounts of data. |
| Approach: | They propose an active learning loop that takes LLMs as annotators and puts them into an active loop to determine what to annotate efficiently. |
| Outcome: | The proposed model outperforms existing models with few-shot performance in two NLP tasks. |
Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems (2025.emnlp-industry)
Copied to clipboard
Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Hejian Sang, Shao Tang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh S. Pillai, Luke Simon, Rahul Mazumder
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications. |
| Approach: | They propose two techniques for training and deploying small language models that deliver high performance for a variety of industry use cases. |
| Outcome: | The proposed techniques retain much of the quality of larger models while reducing training/serving costs and latency. |
A Survey of Post-Training Scaling in Large Language Models (2025.acl-long)
Copied to clipboard
Hanyu Lai, Xiao Liu, Junjie Gao, Jiale Cheng, Zehan Qi, Yifan Xu, Shuntian Yao, Dan Zhang, Jinhua Du, Zhenyu Hou, Xin Lv, Minlie Huang, Yuxiao Dong, Jie Tang
| Challenge: | Large language models (LLMs) have demonstrated proficiency in understanding and generating human natural languages. |
| Approach: | They propose a framework for scaling large language models using supervised fine-tuning, RLxF and test-time compute methodologies. |
| Outcome: | The proposed model can be used to understand and generate human natural languages. |
WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling (2026.findings-acl)
Copied to clipboard
Jiacheng Li, Jianchao Tan, Zhidong Yang, Pingwei Sun, Feiye Huo, Jiayu Qin, Xiangyu Zhang, Maoxin He, Guangming Tan, Weile Jia, Xunliang Cai, Tong Zhao
| Challenge: | Recent advances in training optimization for Transformer-based large language models lack systematic optimization of weight patterns during training. |
| Approach: | They propose a Weight Scaling method that rescales weights while preserving model outputs to improve model training efficiency and model quality. |
| Outcome: | The proposed method significantly improves convergence quality and loss reduction in LLMs with Grouped Query Attention architectures and LoRA fine-tuning tasks. |
Let’s Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are composed of neurons that exhibit diverse behaviors and roles. |
| Approach: | They propose a novel approach that refines the granularity of parameter training down to the individual neuron, enabling a more parameter-efficient fine-tuning model. |
| Outcome: | The proposed approach exceeds the performance of full-parameter fine-tuning and PEFT and provides insights into the analysis of neurons. |
Decoupling Generalization and Adaptation in Meta-Learning for Large Language Models (2026.acl-short)
Copied to clipboard
| Challenge: | Adapting large language models to specific downstream tasks requires multi-step fine-tuning with substantial training data, incurring significant computational overhead. |
| Approach: | They propose a framework that separates learning generalizable initializations and adaptation through dedicated parameter spaces. |
| Outcome: | The proposed framework outperforms existing meta-learning and standard multi-task baselines on common-sense reasoning, mathematics, logic, medical and coding benchmarks. |
DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understanding (2026.eacl-long)
Copied to clipboard
Moulik Choraria, Xinbo Wu, Akhil Bhimaraju, Nitesh Sekhar, Yue Wu, Xu Zhang, Prateek Singhal, Lav R. Varshney
| Challenge: | Recent work shows that hyperscaling of data and parameter count in LLMs is yielding diminishing improvement when weighed against training costs. |
| Approach: | They propose to insert multimodal tokens directly into the middle of the model to bypass the early layers. |
| Outcome: | The proposed method reduces training and inference costs while preserving performance. |
DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Structured pruning reduces model size but often causes uneven degradation across domains, leading to biased performance. |
| Approach: | They propose a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data. |
| Outcome: | Experiments in monolingual and multilingual settings show that the proposed method surpasses similarly sized models in pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. |
Walia-LLM: Enhancing Amharic-LLaMA by Integrating Task-Specific and Generative Datasets (2024.findings-emnlp)
Copied to clipboard
Israel Azime, Atnafu Tonja, Tadesse Belay, Mitiku Yohannes Fuge, Aman Wassie, Eyasu Jada, Yonas Chanie, Walelign Sewunetie, Seid Yimam
| Challenge: | Low-resource languages are left behind due to the unavailability of resources. |
| Approach: | They propose to integrate task-specific and generative datasets to improve language model performance for Amharic by fine-tuning an Amharican instruction fine-to-tuned model. |
| Outcome: | The proposed model shows promising results in different NLP tasks and compares translated instruction datasets with the original model. |