| Challenge: | Recent research shows that neural models are insensitive to word-order perturbations, but other studies suggest that models learn some abstract notion of syntax. |
| Approach: | They develop order-altering perturbations on the order of words, subwords, and characters to analyze their effect on neural models’ performance on language understanding tasks. |
| Outcome: | The proposed models are insensitive to word-order perturbations while the local ordering remains relatively unperturbed. |
Similar Papers
Local Structure Matters Most in Most Languages (2022.aacl-short)
Copied to clipboard
| Challenge: | Recent perturbation studies have found unintuitive results on what does and does not matter when performing Natural Language Understanding (NLU) tasks in English. |
| Approach: | They replicate a study on the importance of local structure and relative unimportance of global structure in a multilingual setting. |
| Outcome: | The proposed model replicates a study on the importance of local structure and relative unimportance of global structure in a multilingual setting. |
Language models and brains align due to more than next-word prediction and word-level information (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models have been shown to significantly predict brain recordings of people comprehending language. |
| Approach: | They propose to use two perturbations to design contrasts that control for different types of information. |
| Outcome: | The proposed model is largely agnostic about the exact linguistic information contained in the conceptual quantities "word-level information" and "multi-word information". |
Do Language Models Exhibit Human-like Structural Priming Effects? (2024.findings-acl)
Copied to clipboard
| Challenge: | a recent exposure to a structure facilitates processing of the same structure, a study finds . structural priming is well attested in humans, for both language production and comprehension . |
| Approach: | They use the structural priming paradigm to investigate where priming effects manifest . they find that rarer elements within a prime increase priming effect . |
| Outcome: | The findings provide an important piece in the puzzle of understanding how properties within their context affect structural prediction in language models. |
It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on societal bias in NLP focuses on race and gender . linguistic background is a unique attribute that has been largely ignored in the field . |
| Approach: | They examine linguistic background to craft plausible adversarial examples that expose biases in popular NLP models. |
| Outcome: | The proposed model improves robustness without sacrificing performance on clean data. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
Towards preserving word order importance through Forced Invalidation (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies show pre-trained language models are insensitive to word order . performance on NLU tasks remains unchanged even after permuting the word . |
| Approach: | They propose a simple approach called Forced Invalidation to force the model to identify permuted sequences as invalid samples. |
| Outcome: | The proposed approach significantly improves the sensitivity of the models to word order on English NLU and QA tasks over BERT-based and attention-based models over word embeddings. |
Language Models as an Alternative Evaluator of Word Order Hypotheses: A Case Study in Japanese (2020.acl-main)
Copied to clipboard
| Challenge: | a method using neural language models (LMs) for analyzing the word order of language is currently lacking. |
| Approach: | They propose a method using neural language models to analyze the word order in Japanese . they test whether there is a parallel between LMs and human word order preference . |
| Outcome: | The proposed method is validated by comparing it with other linguistic studies. |
Perturbation Augmentation for Fairer NLP (2022.emnlp-main)
Copied to clipboard
| Challenge: | Unwanted and often harmful social biases are becoming more salient in NLP research. |
| Approach: | They propose to train a neural perturbation model that rewrites demographic references in text to make them more fair. |
| Outcome: | The proposed model outperforms heuristic alternatives on a large dataset of human annotated text perturbations. |
Interpreting the Robustness of Neural NLP Models to Textual Perturbations (2022.findings-acl)
Copied to clipboard
| Challenge: | Modern Natural Language Processing models are sensitive to input perturbations and their performance can decrease when applied to noisy data. |
| Approach: | They propose to explain the extent to which a model is affected by an unseen textual perturbation by the learnability of the perturbation. |
| Outcome: | The proposed model is better at identifying a perturbation (higher learnability) but worse at ignoring it (lower robustness). |
On the Role of Pre-trained Language Models in Word Ordering: A Case Study with BART (2022.coling-1)
Copied to clipboard
| Challenge: | Existing work uses linear models and neural networks for word ordering, yet pre-trained language models have not been studied in word ordering. |
| Approach: | They propose a constrained language generation task using unordered words as input. |
| Outcome: | The proposed model is able to perform better than existing models and proves to be reliable. |