| Challenge: | Recent advances in NLP have been vulnerable to noisy inputs. |
| Approach: | They propose a contextual text denoising algorithm based on a ready-to-use masked language model that does not require retraining and can be integrated into any NLP system without additional training on paired cleaning training data. |
| Outcome: | The proposed algorithm can correct noise text and improve performance in several downstream tasks. |
Similar Papers
Context Analysis for Pre-trained Masked Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models that learn contextualized word representations from a large un-annotated corpus have become a standard component for many downstream NLP tasks. |
| Approach: | They propose to use a masking and gradient approach to evaluate the impact of context on the word representation. |
| Outcome: | The proposed model architectures are architecture agnostic and gradient based. |
DEEP: DEnoising Entity Pre-training for Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Earlier named entity translation methods focus on phonetic transliteration, which ignores the sentence context for translation. |
| Approach: | They propose a DEnoising Entity Pre-training method that leverages monolingual data and a knowledge base to improve named entity translation accuracy within sentences. |
| Outcome: | The proposed method improves on three language pairs and denoising auto-encoding baselines. |
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that our binary masked language models encode information necessary for solving downstream tasks. |
| Approach: | They propose an efficient method of utilizing pretrained language models where selective binary masks are learned instead of finetuning. |
| Outcome: | Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that the proposed method yields comparable performance to finetuning, but has a much smaller memory footprint when multiple tasks need to be solved. |
Self-Evolution Learning for Discriminative Language Model Pretraining (2023.findings-acl)
Copied to clipboard
| Challenge: | Random masking does not consider the importance of the different words in the sentence meaning, e.g., entity-level masking requires expensive prior knowledge and generally does not use existing model weights. |
| Approach: | They propose a token masking and learning method that uses a random masking strategy to learn the under-explored tokens. |
| Outcome: | The proposed method improves linguistic knowledge learning and generalization on 10 tasks. |
DMLM: Descriptive Masked Language Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | Descriptive Masked Language Modeling (DMLM) is a knowledge-enhanced reading comprehension objective that requires the model to predict the most likely word in a context, being provided with the word’s definition. |
| Approach: | They propose a knowledge-enhanced reading comprehension objective where the model is required to predict the most likely word in a context, being provided with the word’s definition. |
| Outcome: | The proposed model improves on a number of well-established NLU benchmarks and other semantic-focused tasks, e.g., Semantic Role Labeling. |
Improving negation detection with negation-focused pre-training (2022.naacl-main)
Copied to clipboard
| Challenge: | Negation is a common linguistic feature that is crucial in many language understanding tasks. |
| Approach: | They propose a new approach to detect negation in language models using data augmentation and negation masking. |
| Outcome: | The proposed approach improves negation detection performance and generalizability over the strong baseline NegBERT. |
Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | a recent study suggests that masked language models are a useful pre-training technique for natural language processing . a study using mlms pre-trained by a team of researchers has improved performance . |
| Approach: | They propose an alternative to the classic masked language modeling paradigm . they use an unsupervised technique which uses sparse coding to make the prediction possible . |
| Outcome: | The proposed technique improves on pre-trained models compared to vanilla MLM . the proposed model returns distributions over their vocabulary peaking at plausible substitutes . |
BERTective: Language Models and Contextual Information for Deception Detection (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods to classify texts as truthful or deceptive are limited by the context of the text being analyzed. |
| Approach: | They propose to use a corpus of Italian dialogues to classify texts as truthful or deceptive. |
| Outcome: | The proposed models show that not all contexts are equally useful to the task. |
A Cheaper and Better Diffusion Language Model with Soft-Masked Noise (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing diffusion models have limitations in modeling discrete data, e.g., languages . we present a novel diffusion model for language modeling inspired by linguistic features in languages based on iterative denoising . |
| Approach: | They propose a method that iteratively denoises and adds corruptions to the textual data through soft-masking to better noise it. |
| Outcome: | The proposed model achieves better generation quality and lower training cost than current models with better performance. |
Improving Pretraining Techniques for Code-Switched NLP (2023.acl-long)
Copied to clipboard
| Challenge: | Multilingual pretraining models for code-switched inputs are a key component of NLP applications. |
| Approach: | They propose to use masked language modeling techniques to mask code-switched text that are cognizant of language boundaries prior to masking. |
| Outcome: | The proposed techniques improve performance on two downstream tasks, Question Answering (QA) and Sentiment Analysis (SA), compared to standard pretraining techniques. |