The Unreasonable Effectiveness of Easy Training Data for Hard Tasks (2024.acl-long)
Copied to clipboard
| Challenge: | Existing pretrained language models perform well on hard data, but hard data is noisier and costlier to collect. |
| Approach: | They propose to use in-context learning, linear classifier heads, and QLoRA to show that pretrained language models generalize relatively well from easy to hard data. |
| Outcome: | The proposed model generalizes well from easy to hard data even better than oracle models finetuned on hard data. |
Similar Papers
Revisiting Generalization Across Difficulty Levels: It’s Not So Easy (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing research is mixed regarding whether training on easier or harder data leads to better results. |
| Approach: | They examine how well large language models generalize across different task difficulties by using a large dataset and a well-established difficulty metric. |
| Outcome: | The results show that training on hard data can't achieve consistent improvements across the full range of difficulties. |
Data Factors for Better Compositional Generalization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent diagnostic datasets on compositional generalization expose severe problems . state-of-the-art models trained on larger and more general datasets show better generalization ability . |
| Approach: | They conduct an empirical analysis by training Transformer models on a variety of training sets with different data factors including dataset scale, pattern complexity, example difficulty, etc. |
| Outcome: | The proposed model training on larger datasets improves on compositional generalization tasks. |
Guiding Through Complexity: What Makes Good Supervision for Hard Reasoning Tasks? (2025.naacl-long)
Copied to clipboard
| Challenge: | Using weak teacher models to effectively supervise LLMs can improve performance on hard reasoning tasks. |
| Approach: | They propose two data-driven supervision strategies that offer supervision data at different quality levels upon tasks of varying complexity. |
| Outcome: | The proposed methods outperform "perfectly correct" supervision on harder subtasks even when the outcome error rate is high. |
Unveiling the Generalization Power of Fine-Tuned Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional multitasking abilities, but the comprehensive effects of fine-tuning on the LLMs’ generalization ability are not fully understood. |
| Approach: | They conduct extensive experiments across five distinct language tasks on different datasets to investigate whether fine-tuning affects the generalization ability intrinsic to LLMs. |
| Outcome: | The proposed model can generalize to different domains and tasks by integrating the in-context learning strategy during fine-tuning on generation tasks. |
How to Mitigate Overfitting in Weak-to-strong Generalization? (2025.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that weak-to-strong generalization significantly improves PGR compared to naive weak- to-strong . superalignment refers to how humans can align models on tasks beyond human ability to evaluate . |
| Approach: | They propose a framework that elicits the capabilities of strong models through weak supervisors . they propose 'superalignment' to ensure that strong models align with supervisors' intentions . |
| Outcome: | The proposed framework significantly improves quality of supervision signals and quality of input questions compared to naive weak-to-strong generalization . |
EnsemW2S: Enhancing Weak-to-Strong Generalization with Large Language Model Ensembles (2026.findings-acl)
Copied to clipboard
Aakriti Agrawal, Mucong Ding, Chenghao Deng, Zora Che, Arjun Rajaram, Anirudh Satheesh, Bang An, C. Bayan Bruss, John Langford, Furong Huang
| Challenge: | Large Language Models (LLMs) are rapidly approaching and potentially exceeding human-level performance . a novel method aims to improve weak experts' generalization abilities by training them on limited human- level data . |
| Approach: | They propose a method that iteratively combines multiple weak experts to improve their generalization performance by training on limited human-level data. |
| Outcome: | The proposed method improves weak experts' generalization abilities by iterating on weak models and stronger student models. |
Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposes lightweight updates to improve commonsense reasoning models . fine-tuning can cause models to overfit to task-specific data and forget knowledge gained during training . |
| Approach: | They propose to use lightweight models to update pre-trained language models to learn commonsense background knowledge. |
| Outcome: | The proposed models learn from commonsense reasoning datasets, but they are overfitted and limited generalized. |
Generalizing Trust: Weak-to-Strong Trustworthiness in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Recent studies have highlighted weak-to-strong generalization, where a strong model trained only on a weak model’s labels surpasses the weak model in task performance. |
| Approach: | They propose two fundamental fine-tuning strategies that leverage trustworthiness regularization during the fine-uning of the weak model and the weak-to-strong transfer to improve trustworthy. |
| Outcome: | The proposed models show that they can generalize robustness, fairness, and privacy better when trained on weak models than models trained on strong models. |
Memorisation versus Generalisation in Pre-trained Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art pre-trained language models have been shown to memorise facts and perform well with limited amounts of training data. |
| Approach: | They propose to extend pre-trained language models to generalise and memorise facts in noisy and low-resource scenarios. |
| Outcome: | The proposed extension improves performance in low-resource named entity recognition tasks. |
Evaluating Large Language Models on Controlled Generation Tasks (2023.emnlp-main)
Copied to clipboard
Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, Xuezhe Ma
| Challenge: | Recent studies have looked into the ability of large language models in various benchmark tasks, including question generation, reading comprehension, multilingual and etc. However, few studies investigate the controllability of large languages. |
| Approach: | They propose to compare large language models with state-of-the-start finetuned smaller models to find that large language model controls are comparable to smaller models. |
| Outcome: | The proposed model can meet hard constraints and perform better than state-of-the-art models. |