| Challenge: | Existing approaches to computer vision require task-specific modifications and training from scratch. |
| Approach: | They propose a method that can be applied to any task in NLP and propose to open-source it. |
| Outcome: | The proposed method outperforms the state-of-the-art on six text classification tasks, reducing error by 18-24% on majority of datasets. |
Similar Papers
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models fail to generalize to unseen data. |
| Approach: | They propose a framework for robust and efficient fine-tuning for pre-trained models . proposed framework achieves new state-of-the-art performance on a number of NLP tasks . |
| Outcome: | The proposed framework outperforms the state-of-the-art T5 model on GLUE, SNLI, SciTail and ANLI. |
MultiFiT: Efficient Multi-lingual Language Model Fine-tuning (D19-1)
Copied to clipboard
| Challenge: | Pretrained language models require unlabelled data for training, while cross-lingual models underperform on low-resource languages. |
| Approach: | They propose a multi-lingual language model fine-tuning to train and fine- tune language models efficiently in their own language. |
| Outcome: | The proposed method outperforms existing models on two widely used datasets on cross-lingual classification tasks. |
Transfer Learning in Natural Language Processing (N19-5)
Copied to clipboard
| Challenge: | supervised machine learning is based on learning in isolation, a single predictive model for a task using a dataset. |
| Approach: | They present an overview of modern transfer learning methods in natural language processing . they review examples and case studies on how models can be integrated and adapted . |
| Outcome: | The proposed methods improve upon the state-of-the-art on a wide range of NLP tasks. |
An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (N19-1)
Copied to clipboard
| Challenge: | Existing transfer learning methods employ language models pretrained on large generic corpora, but results come at a high computational cost and require task-specific architectures. |
| Approach: | They propose a transfer learning approach that combine a task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. |
| Outcome: | The proposed method surpasses well established transfer learning methods with greater level of complexity on a variety of affective and text classification tasks surpassing well established methods with higher level of difficulty. |
Advancing Language Models through Instruction Tuning: Recent Progress and Challenges (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | tutorial addresses three critical questions within the field of instruction tuning: (1) What are the current focal points in instruction tuning research? (2) What are best practices in training an instruction-following model? (3) What new challenges have emerged? |
| Approach: | This tutorial presents a systematic overview of recent advances in instruction tuning. |
| Outcome: | The tutorial covers different stages in model training: supervised fine-tuning, preference optimization, and reinforcement learning. |
ULLME: A Unified Framework for Large Language Model Embeddings with Generation-Augmented Learning (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Existing frameworks for large language model embeddings have limited support for only a limited range of architectures and fine-tuning strategies. |
| Approach: | They propose a framework that enables bidirectional attention across various LLMs and supports a range of fine-tuning strategies. |
| Outcome: | The proposed framework enables bidirectional attention across various LLMs and supports a range of fine-tuning strategies. |
Modular and Parameter-Efficient Fine-Tuning for NLP Models (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | State-of-the-art language models in NLP perform best when fine-tuned even on small datasets. |
| Approach: | They provide an overview of parameter-efficient fine-tuning methods and highlight similarities and differences . they highlight benefits and usage scenarios of a neglected property of parameter efficient models . |
| Outcome: | This paper provides an overview of parameter-efficient fine-tuning methods . it highlights similarities and differences by presenting them in a unified view . |
Language Models for Text Classification: Is In-Context Learning Enough? (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing research on text classification models with prompts is limited in scale and lacks understanding of how these methods compare to more established methods. |
| Approach: | They compare the performance of large and smaller language models with prompts to achieve state-of-the-art performance in many NLP tasks. |
| Outcome: | The proposed models outperform the more standard approaches in binary, multiclass, and multilabel tasks in a large scale evaluation of 16 text classification datasets. |
Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide (2026.tacl-1)
Copied to clipboard
| Challenge: | Pre-trained language models provide strong foundations, but effective adaptation under data scarcity requires efficient and efficient fine-tuning techniques. |
| Approach: | They propose to review parameter-efficient fine-tuning techniques that lower training and deployment costs and domain and cross-lingual adaptation methods for both encoder and decoder models. |
| Outcome: | The proposed techniques lower training and deployment costs, domain and cross-lingual adaptation methods, and model specialization strategies. |
On the Importance of Effectively Adapting Pretrained Language Models for Active Learning (2022.acl-short)
Copied to clipboard
| Challenge: | Recent active learning approaches in NLP use off-the-shelf pretrained language models (LMs) . a poor training strategy can be catastrophic for AL, authors argue . |
| Approach: | They propose to first adapt the pretrained LM to the target task and then use it for AL. |
| Outcome: | The proposed approach provides substantial data efficiency improvements compared to the standard fine-tuning approach. |