Imbalanced Gradients in RL Post-Training of Multi-Task LLMs (2026.findings-eacl)
Copied to clipboard
Runzhe Wu, Ankur Samanta, Ayush Jain, Scott Fujimoto, Jeongyeol Kwon, Ben Kretzu, Youliang Yu, Kaveh Hassani, Boris Vidolov, Yonathan Efroni
| Challenge: | Large-gradient tasks can achieve similar or even much lower learning gains than small-grading ones. |
| Approach: | They show that large-gradient tasks can achieve lower learning gains than small-grading ones . large-grade tasks can accomplish similar or even lower learning gain than small grade ones if they are large . |
| Outcome: | The proposed approach fails when certain tasks produce larger gradients . Large-gradient tasks can achieve lower learning gains than small-gradent ones . |
Similar Papers
DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Structured pruning reduces model size but often causes uneven degradation across domains, leading to biased performance. |
| Approach: | They propose a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data. |
| Outcome: | Experiments in monolingual and multilingual settings show that the proposed method surpasses similarly sized models in pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. |
Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve? (2024.emnlp-main)
Copied to clipboard
| Challenge: | In the last decade, the generalization and adaptation abilities of deep learning models were evaluated on fixed training and test distributions. |
| Approach: | They propose to train large language models on unlabeled text corpora and train them online. |
| Outcome: | The proposed model training on a text domain could degrade its perplexity on the test portion of the same domain. |
A Survey of Post-Training Scaling in Large Language Models (2025.acl-long)
Copied to clipboard
Hanyu Lai, Xiao Liu, Junjie Gao, Jiale Cheng, Zehan Qi, Yifan Xu, Shuntian Yao, Dan Zhang, Jinhua Du, Zhenyu Hou, Xin Lv, Minlie Huang, Yuxiao Dong, Jie Tang
| Challenge: | Large language models (LLMs) have demonstrated proficiency in understanding and generating human natural languages. |
| Approach: | They propose a framework for scaling large language models using supervised fine-tuning, RLxF and test-time compute methodologies. |
| Outcome: | The proposed model can be used to understand and generate human natural languages. |
Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classifications (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies suggest using large language models to make tabular classifications . however, LLMs have been shown to exhibit harmful social biases based on stereotypes and inequalities present in society. |
| Approach: | They propose to use large language models to make tabular classifications . they show that LLMs inherit biases from their training data . |
| Outcome: | The proposed models exhibit harmful biases that reflect stereotypes and inequalities in society. |
How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients (2026.acl-long)
Copied to clipboard
| Challenge: | Spectral properties of low/high-quality instruction and reasoning data are used to explain finetuning dynamics in large language models. |
| Approach: | They propose to analyze layer-wise gradients induced by low/high-quality instruction and reasoning data for LLM post-training. |
| Outcome: | The results show that higher-quality data are associated with lower nuclear norms and higher effective ranks. |
Analyzing Effects of Learning Downstream Tasks on Moral Bias in Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models replicate and perpetuate social biases . pre-existing moral bias may be mitigated or amplified even when presented with opposing views . |
| Approach: | They develop methods to assess the agreement of LMs to explicit codified norms . they find that introducing downstream tasks may lead to unexpected inconsistencies . |
| Outcome: | The proposed model can be used to improve morality in data-scarce tasks. |
What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective (2025.acl-long)
Copied to clipboard
| Challenge: | Xu et al., 2024) study shows that slow thinking can distinguish correct and irrelevant reasoning paths. |
| Approach: | They investigate how fast vs. slow thinking affects layer-wise gradients in large language models . they find that slow thinking can distinguish correct and irrelevant reasoning paths . |
| Outcome: | The results show that slow thinking can distinguish correct and irrelevant reasoning paths. |
Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained large language models retain task-specific knowledge, but where and to what extent they retain it remains unexplored. |
| Approach: | They investigate the task-specific information encoded in pre-trained LLMs and the effects of instruction tuning on their representations across over 60 NLP tasks. |
| Outcome: | The results show that pre-trained models retain task-specific knowledge . some tasks are already encoded in pre-train models, but others benefit from instruction tuning. |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |
Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible (2026.eacl-long)
Copied to clipboard
| Challenge: | linguists have discovered patterns which hold across virtually all known natural languages . lingulists are able to learn languages by comparing their learning curves to those of humans . |
| Approach: | They compare LLM learning curves on existing and "impossible" datasets . they find that GPT-2 learns each language and its impossible counterpart equally easily . |
| Outcome: | The proposed model learns each language and its impossible counterpart equally easily, the study shows . the study also shows that the proposed model does not provide any kind of separation between the possible and the impossible . |