ConvFiT: Conversational Fine-Tuning of Pretrained Language Models (2021.emnlp-main)
Copied to clipboard
Ivan Vulić, Pei-Hao Su, Samuel Coope, Daniela Gerz, Paweł Budzianowski, Iñigo Casanueva, Nikola Mrkšić, Tsung-Hsien Wen
| Challenge: | Existing Transformer-based language models (LMs) are not effective as sentence encoders when used off-the-shelf. |
| Approach: | They propose a method which turns a pretrained LM into a universal conversational encoder and task-specialised sentence encoder. |
| Outcome: | The proposed framework achieves state-of-the-art ID performance across the board with particular gains in the most challenging, few-shot setups. |
Similar Papers
LexFit: Lexical Fine-Tuning of Pretrained Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Transformer-based language models implicitly store a wealth of lexical semantic knowledge, but it is non-trivial to extract that knowledge effectively from their parameters. |
| Approach: | They propose to expose and enrich lexical knowledge from transformer-based language models to serve as effective decontextualized word encoders even when fed input words "in isolation" |
| Outcome: | The proposed model outperforms standard static WEs and vanilla LMs in lexical tasks over four established tasks in 8 languages. |
ConveRT: Efficient and Accurate Conversational Representations from Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | ConveRT is a pretraining framework for conversational AI that is computationally heavy, slow, and expensive to train. |
| Approach: | They propose a pretraining framework for conversational tasks that is efficient, lightweight, and inexpensive. |
| Outcome: | The proposed model achieves state-of-the-art performance across widely established responses . it trains substantially faster than existing state- of-the art models . |
CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing code LLMs adopt a specific architecture or rely on a unified encoder-decoder network for downstream tasks, lacking flexibility to operate in the optimal architecture for a particular task. |
| Approach: | They propose to initialize code LLMs with frozen off-the-shelf LLM and explore instruction-tuning to align with natural language instructions. |
| Outcome: | The proposed model outperforms open-source LLMs on 20 code-related benchmarks. |
Lightweight Transformers for Conversational AI (2022.naacl-industry)
Copied to clipboard
| Challenge: | Commercial dialogue systems typically require a small footprint and fast execution time, but recent trends are in the other direction, resulting in difficulties in model deployment. |
| Approach: | They build Transformer-based Language Models from scratch on large corpora of conversational data and compare their performance against BERT and other strong baselines on dialogue probing tasks. |
| Outcome: | The proposed model outperforms existing models on dialogue probing tasks and can be fine-tuned on a single consumer GPU card. |
Infusing Finetuning with Semantic Dependencies (2021.tacl-1)
Copied to clipboard
| Challenge: | Several diagnostics help to localize the benefits of our approach. |
| Approach: | They apply convolutional graph encoders to integrate semantic parses into task-specific finetuning. |
| Outcome: | The proposed approach yields benefits to natural language understanding (NLU) tasks in the GLUE benchmark. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
MultiFiT: Efficient Multi-lingual Language Model Fine-tuning (D19-1)
Copied to clipboard
| Challenge: | Pretrained language models require unlabelled data for training, while cross-lingual models underperform on low-resource languages. |
| Approach: | They propose a multi-lingual language model fine-tuning to train and fine- tune language models efficiently in their own language. |
| Outcome: | The proposed method outperforms existing models on two widely used datasets on cross-lingual classification tasks. |
A Million Tweets Are Worth a Few Points: Tuning Transformers for Customer Service Tasks (2021.naacl-main)
Copied to clipboard
| Challenge: | In domain-specific customer service applications, many companies struggle to deploy advanced NLP models due to the limited availability of and noise in their datasets. |
| Approach: | They analyze customer service conversations on a multilingual social media corpus and compare different approaches to pretraining and finetuning on different end tasks. |
| Outcome: | The proposed model improves performance on multilingual social media data, especially in non-English settings. |
Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction (P19-1)
Copied to clipboard
| Challenge: | Current relation extraction methods suffer from noisy labels and incomplete knowledge base information. |
| Approach: | They propose a pre-trained language model that captures semantic and syntactic features and a significant amount of “common-sense” knowledge. |
| Outcome: | The proposed model achieves state-of-the-art AUC score of 0.422 on the NYT10 dataset and performs especially well at higher recall levels. |
Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained Language Models (PLMs) are advanced but data labels are noisy due to the complex annotation process. |
| Approach: | They propose a framework for fine-tuning PLMs using noisy labels that incorporates guidance from Large Language Models like ChatGPT. |
| Outcome: | Experiments on synthetic and real-world noisy datasets show that the proposed framework outperforms the state-of-the-art framework. |