Label Semantic Aware Pre-training for Few-shot Text Classification (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models for text classification use label semantics but few studies have attempted to give models access to informative representations of labels. |
| Approach: | They propose to use label semantics to train generative models by performing secondary pre-training on labeled sentences from a variety of domains. |
| Outcome: | The proposed approach improves generalization and data efficiency of text classification systems while maintaining comparable performance to state-of-the-art models. |
Similar Papers
Don’t Miss the Labels: Label-semantic Augmented Meta-Learner for Few-Shot Text Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on building a meta-learner from input text but ignore abundant semantic information beneath class labels. |
| Approach: | They propose a framework to make full use of label semantics in few-shot text classification systems. |
| Outcome: | The proposed framework can be plugged into the existing few-shot text classification system. |
Label Agnostic Pre-training for Zero-shot Text Classification (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to text classification assume a fixed set of labels . however, in real-world applications, there exists an infinite label space for describing a given text . |
| Approach: | They propose two new methods that inject aspect-level understanding into pre-trained models at train time to improve zero-shot generalization. |
| Outcome: | The proposed methods improve zero-shot generalization on a set of challenging datasets. |
Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to few-shot text classification require domain expertise and an understanding of the language model's abilities to define the mapping between words and labels. |
| Approach: | They propose a method that converts textual inputs to cloze questions that contain some form of task description and processes them with a pretrained language model to map the predicted words to labels. |
| Outcome: | The proposed approach performs almost as well as hand-crafted label-to-word mappings for a number of tasks with small amounts of training data. |
The Benefits of Label-Description Training for Zero-Shot Text Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models have improved zero-shot text classification by allowing the transfer of semantic knowledge from the training data to classify among specific label sets in downstream tasks. |
| Approach: | They propose to use a small finetuning dataset to describe the labels for a task and to use it to further improve zero-shot accuracies. |
| Outcome: | The proposed model is more accurate than zero-shot by 17-19% absolute across topic and sentiment datasets and more robust to choices required for zero- shot classification. |
Effectiveness of Pre-training for Few-shot Intent Classification (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing paradigms further pre-train language models such as BERT on vast amount of unlabeled corpus, but we find it highly effective and efficient to simply fine-tune BERT with roughly 1,000 labeled utterances from public datasets. |
| Approach: | They propose to fine-tune BERT with a small set of labeled utterances from public datasets to achieve a pre-trained model based on a set of 1,000 labeles. |
| Outcome: | The proposed model can outperform existing models on domains with very different semantics on novel domains. |
Combining Unsupervised Pre-training and Annotator Rationales to Improve Low-shot Text Classification (D19-1)
Copied to clipboard
| Challenge: | supervised learning models perform poorly at low-shot tasks for which little labeled data is available for training. |
| Approach: | They propose to combine a bag-of-words embedding approach and a context-aware method to improve low-shot text classification. |
| Outcome: | The proposed method improves low-shot text classification with pre-training and rationales . the simple bag-of-words approach is the clear top performer when there are few training instances or less . |
Pre-training Intent-Aware Encoders for Zero- and Few-Shot Intent Classification (2023.emnlp-main)
Copied to clipboard
Mujeen Sung, James Gung, Elman Mansimov, Nikolaos Pappas, Raphael Shu, Salvatore Romeo, Yi Zhang, Vittorio Castelli
| Challenge: | Existing methods for IC training do not provide sufficient examples for each intent . a novel pre-training method is proposed to provide a better understanding of intents . |
| Approach: | They propose a method that uses contrastive learning with intent psuedo-labels to produce embeddings that are well-suited for IC tasks. |
| Outcome: | The proposed method achieves 5.4% and 4.0% higher accuracy than the current state-of-the-art method on four IC datasets. |
Label Semantics for Few Shot Named Entity Recognition (2022.findings-acl)
Copied to clipboard
Jie Ma, Miguel Ballesteros, Srikanth Doss, Rishita Anubhai, Sunil Mallya, Yaser Al-Onaizan, Dan Roth
| Challenge: | Named entity recognition (NER) is a fundamental natural language understanding task that requires large amounts of high quality annotated in-domain data. |
| Approach: | They propose a neural architecture that leverages the semantic information in the names of the labels to give the model additional signal and enriched priors. |
| Outcome: | The proposed model is especially effective in low resource settings. |
Label-Aware Automatic Verbalizer for Few-Shot Text Classification in Mid-To-Low Resource Languages (2024.acl-srw)
Copied to clipboard
| Challenge: | Prompt-based learning has shown its effectiveness in few-shot text classification. |
| Approach: | They propose a prompt-based learning verbalizer that automatically selects a word to represent each class . they use the label name along with the conjunction "and" to induce the model to generate more effective words for the verbaliser. |
| Outcome: | The proposed method outperforms existing verbalizers on four Southeast Asian languages. |
Task-Aware Representation of Sentences for Generic Text Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to text classification use a transformer architecture with a linear layer on top. |
| Approach: | They propose a transformer-based approach that outputs a class distribution for a given prediction problem. |
| Outcome: | The proposed model outperforms existing approaches on small training data and can learn to predict new classes even with no training examples. |