| Challenge: | Semi-supervised learning (SSL) is a popular setting to make use of unlabelled data . Currently, there are two popular approaches to make effective use of the unlabelled datasets . |
| Approach: | They compare semi-supervised learning (SSL) and task-adaptive pre-training (TAPT) they find TAPT is a stronger and more robust SSL learner, even when using just a few hundred unlabelled samples . |
| Outcome: | The proposed methods improve model performance across different NLP tasks and data sizes. |
Similar Papers
Industry Scale Semi-Supervised Learning for Natural Language Understanding (2021.naacl-industry)
Copied to clipboard
| Challenge: | Obtaining human annotation is expensive and time-consuming process. |
| Approach: | They propose a semi-supervised learning pipeline which leverages millions of unlabeled examples to improve natural language understanding tasks. |
| Outcome: | The proposed pipeline can be used to improve natural language understanding tasks. |
Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Task-adaptive pre-training (TAPT) and Self-training can be complementary with simple TFS protocol. |
| Approach: | They propose to use task-adaptive pre-training and self-training to combine TAPT and ST with a simple TFS protocol to achieve strong combined gains across six datasets. |
| Outcome: | The proposed method can achieve strong combined gains across six datasets covering sentiment classification, paraphrase identification, natural language inference, named entity recognition and dialogue slot classification. |
Learning to Infer from Unlabeled Data: A Semi-supervised Learning Approach for Robust Natural Language Inference (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Semi-supervised learning (SSL) is a popular technique for reducing the reliance on human annotations for NLI tasks. |
| Approach: | They propose a way to incorporate unlabeled data into semi-supervised learning (SSL) using a conditional language model, they propose to generate hypotheses for unlabed sentences . |
| Outcome: | The proposed framework significantly improves the performance of four NLI datasets in low-resource settings. |
Leveraging Training Dynamics and Self-Training for Text Classification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Semi-supervised learning (SSL) is a promising technique for improving deep learning models when training data is scarce. |
| Approach: | They propose a semi-supervised learning approach that leverages training dynamics of unlabeled data. |
| Outcome: | The proposed method achieves an average increase in F1 score of 3.5% over baselines in low resource settings. |
Semi-Supervised Lifelong Language Learning (2022.findings-emnlp)
Copied to clipboard
Yingxiu Zhao, Yinhe Zheng, Bowen Yu, Zhiliang Tian, Dongkyu Lee, Jian Sun, Yongbin Li, Nevin L. Zhang
| Challenge: | Existing methods to learn languages only focus on supervised learning, and unlabeled data is underexplored. |
| Approach: | They propose a semi-supervised lifelong language learning setting where a model learns sequentially arriving language tasks with both labeled and unlabeled data. |
| Outcome: | The proposed model outperforms baseline models on various language tasks and is effective and superior to existing models. |
Self-supervised Representation Learning for Speech Processing (2022.naacl-tutorials)
Copied to clipboard
Hung-yi Lee, Abdelrahman Mohamed, Shinji Watanabe, Tara Sainath, Karen Livescu, Shang-Wen Li, Shu-wen Yang, Katrin Kirchhoff
| Challenge: | Self-supervised representation learning (SSL) uses proxy supervised learning tasks to obtain training data from unlabeled corpora. |
| Approach: | They propose to survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing to scale up current machine learning technologies. |
| Outcome: | The proposed tutorial is highly relevant to the special theme of ACL about language diversity. |
Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification (2021.eacl-main)
Copied to clipboard
| Challenge: | Semi-supervised learning and multilingual pretraining have been shown to be effective for task-specific labelled data shortages. |
| Approach: | They propose to combine semi-supervised deep generative models and multi-lingual pretraining to form a pipeline for document classification task. |
| Outcome: | The proposed method outperforms state-of-the-art models in low-resource settings across several languages and outperformed existing models in English. |
Self-supervised Regularization for Text Classification (2021.tacl-1)
Copied to clipboard
| Challenge: | Text classification models are prone to overfitting when limited texts are available for training. |
| Approach: | They propose a data-dependent regularization approach based on self-supervised learning . they define auxiliary tasks on input data without using human-provided labels . |
| Outcome: | Experiments on 17 text classification datasets demonstrate the effectiveness of the proposed method. |
Combining Weakly Supervised ML Techniques for Low-Resource NLU (2021.naacl-industry)
Copied to clipboard
| Challenge: | Recent advances in transfer learning have improved the performance of virtual assistants . however, meager training data is often a key bottleneck in creating voice-enabled applications . |
| Approach: | They propose to use unsupervised and semi-supervised techniques to improve NLU accuracy . they incorporate anonymized, unlabeled and automatically transcribed user utterances into training . |
| Outcome: | The proposed methods improve NLU accuracy in low-resource settings by integrating unsupervised and SSL techniques. |
An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to learning data representations using contrastive learning perform data augmentation and contrastive training separately. |
| Approach: | They propose a framework that performs data augmentation and contrastive learning end-to-end . they propose to combine data augmented with text encoders to optimize for contrastive training . |
| Outcome: | Experiments on GLUE and Gururangan datasets show the proposed framework is effective in NLP. |