| Challenge: | a thesis proposal advocates for the crucial role of cross-task generalization in NLP systems. |
| Approach: | They propose to benchmark cross-task generalization abilities with diverse NLP tasks . they also propose to develop model architectures for improving cross- task generalization . |
| Outcome: | This paper compares cross-task generalization abilities with diverse NLP tasks . it also analyzes and predicts the generalization landscape of current state-of-the-art large language models . |
Similar Papers
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2022.acl-long)
Copied to clipboard
| Challenge: | Despite the success of supervised learning, models often struggle with generalization across tasks. |
| Approach: | They propose to use crowdsourcing instructions to build a model that learns a new task by understanding the human-readable instructions that define it. |
| Outcome: | The proposed model can learn from seen tasks and generalize to unseen tasks given its natural crowdsourcing instructions. |
Exploring Compositional Generalization of Large Language Models (2024.naacl-srw)
Copied to clipboard
| Challenge: | a recent study has found that large language models can generalize compositional instructions from simple instructions to complex ones. |
| Approach: | They study the generalization ability of large language models with respect to compositional instructions . they first construct a dataset with the help of ChatGPT guided by the self-instruct technique . |
| Outcome: | The proposed model can generalize from simple instructions to more intricate ones, the authors show . their results show that training LLMs on higher-order compositional instructions improves performance on lower-order ones, but not on higher order ones. |
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022.emnlp-main)
Copied to clipboard
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, Xudong Shen
| Challenge: | a benchmark of 1,616 diverse NLP tasks and their expert-written instructions is used to test generalization of models to unseen tasks . a recent study shows that instruction-following models outperform instruction-based models by over 9% . |
| Approach: | They build a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. |
| Outcome: | The proposed model outperforms existing instruction-following models by over 9% on the benchmark despite being smaller. |
Exploring and Predicting Transferability across NLP Tasks (2020.emnlp-main)
Copied to clipboard
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, Mohit Iyyer
| Challenge: | Recent advances in NLP demonstrate the effectiveness of training large-scale language models and transferring them to downstream tasks. |
| Approach: | They conduct an extensive study of the transferability between 33 NLP tasks across three broad classes of problems. |
| Outcome: | The proposed model can improve performance even with low-data source tasks that differ substantially from the target task. |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |
Evaluating Large Language Models on Controlled Generation Tasks (2023.emnlp-main)
Copied to clipboard
Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, Xuezhe Ma
| Challenge: | Recent studies have looked into the ability of large language models in various benchmark tasks, including question generation, reading comprehension, multilingual and etc. However, few studies investigate the controllability of large languages. |
| Approach: | They propose to compare large language models with state-of-the-start finetuned smaller models to find that large language model controls are comparable to smaller models. |
| Outcome: | The proposed model can meet hard constraints and perform better than state-of-the-art models. |
LLM-driven Instruction Following: Progresses and Concerns (2023.emnlp-tutorial)
Copied to clipboard
| Challenge: | a tutorial on task instruction is aimed at researchers and practitioners interested in NLP generalization . labeled examples are unlikely to be available in large numbers or do not exist . |
| Approach: | This tutorial will examine the progress of natural language processing (NLP) using labeled examples. authors propose that task instructions act as a novel resource for supervision. |
| Outcome: | This tutorial aims to answer questions about instruction-driven NLP . it focuses on the use of task instructions in a low-shot scenario . |
Unveiling the Generalization Power of Fine-Tuned Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional multitasking abilities, but the comprehensive effects of fine-tuning on the LLMs’ generalization ability are not fully understood. |
| Approach: | They conduct extensive experiments across five distinct language tasks on different datasets to investigate whether fine-tuning affects the generalization ability intrinsic to LLMs. |
| Outcome: | The proposed model can generalize to different domains and tasks by integrating the in-context learning strategy during fine-tuning on generation tasks. |
A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on syntactic knowledge models has not provided a clear picture of the properties required to produce proper syntaktic generalizations. |
| Approach: | They propose to evaluate syntactic knowledge of language models by varying model architectures . they find substantial differences in syntaktic generalization performance by model architecture . |
| Outcome: | The proposed model architectures outperform other architectures on a set of 34 English-language syntactic test suites. |
Blackbird language matrices (BLM), a new task for rule-like generalization in neural networks: Can Large Language Models pass the test? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate large language models for generalization lack generalization ability . current methods for evaluating LLMs are based on tests of human intelligence . |
| Approach: | They propose to use a language task to evaluate large language models' generalisation ability . they propose to ask LLMs to solve simple variants of the RAVEN IQ test . |
| Outcome: | The proposed task can be used to evaluate the generalisation ability of large language models . it shows that current generative models can handle the task in the sense that they understand instructions . |