| Challenge: | In this paper, we introduce Attributive Masking Learning (AML), a method designed for explaining language model predictions by learning input masks. |
| Approach: | They introduce a method for explaining language model predictions by learning input masks and ensuring a significant change in the model's explanation when applying the inverse mask to the input. |
| Outcome: | The proposed method outperforms state-of-the-art explanation methods on multiple benchmarks. |
Similar Papers
Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for improving model interpretability require prior information or human annotations as additional inputs. |
| Approach: | They propose a variational word mask method to automatically learn task-specific important words and reduce irrelevant information on classification, which ultimately improves model interpretability. |
| Outcome: | The proposed method improves model prediction accuracy and interpretability on seven datasets. |
Mask-Predict: Parallel Decoding of Conditional Masked Language Models (D19-1)
Copied to clipboard
| Challenge: | a masked language model is used to train a model to predict subsets of mangled words . a parallel decoding algorithm can be used to generate translations in a constant number of iterations. |
| Approach: | They propose a model and a parallel decoding algorithm which train a machine to predict any subset of target words . they introduce conditional masked language models (CMLMs) which are trained with a mangled language model objective . |
| Outcome: | The proposed model improves state-of-the-art performance levels for non-autoregressive and parallel decoding models by over 4 BLEU on average. |
Choosing What to Mask: More Informed Masking for Multimodal Machine Translation (2023.acl-srw)
Copied to clipboard
| Challenge: | Pre-trained language models have achieved remarkable results on several NLP tasks. |
| Approach: | They propose three new masking strategies for cross-lingual visual pre-training that focus on learning different linguistic patterns. |
| Outcome: | The proposed methods outperform the baseline model and achieve state-of-the-art accuracy on the Portuguese-English MMT task. |
DMLM: Descriptive Masked Language Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | Descriptive Masked Language Modeling (DMLM) is a knowledge-enhanced reading comprehension objective that requires the model to predict the most likely word in a context, being provided with the word’s definition. |
| Approach: | They propose a knowledge-enhanced reading comprehension objective where the model is required to predict the most likely word in a context, being provided with the word’s definition. |
| Outcome: | The proposed model improves on a number of well-established NLU benchmarks and other semantic-focused tasks, e.g., Semantic Role Labeling. |
Enabling Autoregressive Models to Fill In Masked Tokens (2026.findings-eacl)
Copied to clipboard
| Challenge: | Autoregressive (AR) and masked language modeling (MLM) models are incapable of mucked infilling, which is the ability to predict mangled tokens between past and future context. |
| Approach: | They propose a method that leverages the strengths of autoregressive and masked language modeling to achieve state-of-the-art mucked infilling performance. |
| Outcome: | The proposed approach outperforms existing methods on masked infilling tasks. |
A Better Way to Do Masked Language Model Scoring (2023.acl-short)
Copied to clipboard
| Challenge: | Existing methods to estimate log-likelihood of sentences are not available for masked language models . et al., 2020: e.on.org/sci/sql/src/spll-word-l2r estimates sentence pseudo-log-likellihoods for a given sentence under autoregressive models are difficult to perform on controlled benchmarks. |
| Approach: | They propose to estimate sentence pseudo-log-likelihood scores by successively masking each sentence token, retrieving its score using the rest of the sentence as context, and summing the resulting values. |
| Outcome: | The proposed method outperforms both the original and the adapted method on masked language models. |
Contextual Representation Learning beyond Masked Language Modeling (2022.acl-long)
Copied to clipboard
| Challenge: | masked language models adopt sampled embeddings as anchors to estimate and inject contextual semantics to representations. |
| Approach: | They propose a representation learning approach that uses embeddings as anchors to model contextual representations. |
| Outcome: | The proposed model achieves 5x speedup and 1.2 points average improvement over MLM. |
Extrapolating Multilingual Understanding Models as Multilingual Generators (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual understanding models are not capable of generating high-quality text compared with decoder-based causal language models. |
| Approach: | They propose a method to adapt a multilingual encoder to a language generator with a small number of additional parameters. |
| Outcome: | The proposed approach outperforms initialization-based methods with 9.4 BLEU on machine translation, 8.1 Rouge-L on question generation, and 5.5 METEOR on story generation. |
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies suggest that pretrained language models acquire useful inductive biases through masks that implicitly act as cloze reductions for downstream tasks. |
| Approach: | They construct cloze-like masks using task-specific lexicons to explain their results . they show that the majority of performance gains come from generic masks that are not associated with the lexical . |
| Outcome: | The proposed method outperforms a classic method for unsupervised parsing. |
Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models excel in natural language understanding (NLU) tasks. |
| Approach: | They propose to apply layer-dependent removal of the causal mask (CM) during LLM fine-tuning to improve SL performance. |
| Outcome: | The proposed approach outperforms state-of-the-art SL models on IE tasks, while achieving state- of-the art results is unclear. |