WALNUT: A Benchmark on Semi-weakly Supervised Learning for Natural Language Understanding (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on weak supervision for NLU focus on a specific task or simulate weak supervision signals from ground-truth labels. |
| Approach: | They propose a benchmark to advocate and facilitate research on weak supervision for NLU . they use document-level and token-level prediction tasks as examples . |
| Outcome: | The proposed benchmark advocates and facilitates research on weak supervision for NLU tasks. |
Similar Papers
Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural natural language generation and understanding models are data-hungry and require massive amounts of annotated data to be competitive. |
| Approach: | They propose a framework that automatically synthesizes weak labels from large-scale weakly-labeled data with a fine-tuned GPT-2 and adapts parameter updates to the models according to the estimated label-quality. |
| Outcome: | The proposed framework outperforms benchmark systems on the E2E and Weather datasets when 100% of the training data is used. |
Combining Weakly Supervised ML Techniques for Low-Resource NLU (2021.naacl-industry)
Copied to clipboard
| Challenge: | Recent advances in transfer learning have improved the performance of virtual assistants . however, meager training data is often a key bottleneck in creating voice-enabled applications . |
| Approach: | They propose to use unsupervised and semi-supervised techniques to improve NLU accuracy . they incorporate anonymized, unlabeled and automatically transcribed user utterances into training . |
| Outcome: | The proposed methods improve NLU accuracy in low-resource settings by integrating unsupervised and SSL techniques. |
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021.naacl-main)
Copied to clipboard
| Challenge: | Evaluation for many natural language understanding (NLU) tasks is broken due to unreliable and biased systems scoring so high on standard benchmarks. |
| Approach: | They argue that current benchmarks fail at four criteria for evaluation . they argue that adversarial data collection does not address the causes of failures . |
| Outcome: | The proposed frameworks fail at four criteria, and adversarial data collection does not address the causes of these failures, the authors argue . restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, reliability with which they are annotated, their size, and the ways they handle social bias. |
Graph-Based Semi-Supervised Learning for Natural Language Understanding (D19-53)
Copied to clipboard
| Challenge: | Semi-supervised learning is an efficient method to augment training data from unlabeled data. |
| Approach: | They propose semi-supervised learning models and their inductive variants for NLU and use them to find similar utterances and construct a graph. |
| Outcome: | The proposed model improves the error rate of the model by 5% using publicly available NLU data and models. |
Self-Training with Weak Supervision (2021.naacl-main)
Copied to clipboard
| Challenge: | State-of-the-art deep neural networks require large amounts of labeled training data that is expensive to obtain or not available for many tasks. |
| Approach: | They propose a weak supervision framework that leverages all available data for a given task . they leverage task-specific unlabeled data through self-training with a model that predicts pseudo-labels for instances that may not be covered by weak rules . |
| Outcome: | The proposed framework improves on state-of-the-art datasets on six benchmark tasks. |
Adversarial NLI: A New Benchmark for Natural Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | a new large-scale NLI benchmark dataset is presented to test models on a variety of popular NLIs. |
| Approach: | They propose a large-scale NLI benchmark dataset that is iteratively compared with a human-and-model-in-the-loop procedure. |
| Outcome: | The proposed method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate. |
NLU++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue (2022.findings-naacl)
Copied to clipboard
| Challenge: | NLU++ provides a more challenging evaluation environment for dialogue NLU models . Typical ToD systems still rely on a modular design . |
| Approach: | They propose to use NLU++ to provide a more challenging evaluation environment for dialogue NLU models. |
| Outcome: | The proposed dataset improves existing datasets and provides a much more challenging evaluation environment for dialogue NLU models. |
Weakly Supervised Word Segmentation for Computational Language Documentation (2022.acl-long)
Copied to clipboard
| Challenge: | a recent paper aims to improve the effectiveness of unsupervised language analysis techniques in low resource settings. |
| Approach: | They propose to use a weak supervision to improve linguistic segmentation in low resource languages . they propose to provide linguists with LTs that can be used to create interactive annotation tools . |
| Outcome: | The proposed models can be used to improve the quality of language segmentation in low resource languages. |
Weaker Than You Think: A Critical Look at Weakly Supervised Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Weakly supervised learning is a popular approach for training machine learning models in low-resource settings. |
| Approach: | They propose to use weakly supervised learning to train models with noisy labels from weak sources instead of collecting expensive human annotations. |
| Outcome: | The proposed methods outperform weakly supervised methods on various NLP datasets and tasks on the test sets. |
Industry Scale Semi-Supervised Learning for Natural Language Understanding (2021.naacl-industry)
Copied to clipboard
| Challenge: | Obtaining human annotation is expensive and time-consuming process. |
| Approach: | They propose a semi-supervised learning pipeline which leverages millions of unlabeled examples to improve natural language understanding tasks. |
| Outcome: | The proposed pipeline can be used to improve natural language understanding tasks. |