Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs (D19-1)
Copied to clipboard
Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, Samuel R. Bowman
| Challenge: | Recent work evaluating sentence representation models' knowledge of grammar has been slower to emerge. |
| Approach: | They propose five experimental methods inspired by prior work evaluating pretrained sentence representation models to examine their grammatical knowledge. |
| Outcome: | The proposed methods show that the model has significant knowledge of the licensing environment but its success varies widely across different methods. |
Similar Papers
Language Models Use Monotonicity to Assess NPI Licensing (2021.findings-acl)
Copied to clipboard
| Challenge: | Neural language models (LMs) have become powerful approximators of human language . fewer studies have been done on what kind of formal semantic features are encoded by LMs . |
| Approach: | They propose a series of experiments that investigate the semantic knowledge of language models . they use diagnostic classifiers, linguistic acceptability tasks and a ranking method to investigate the models' inner workings. |
| Outcome: | The proposed method can be applied to LMs trained on filtered corpora and gain stronger insights into their generalizations. |
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Pretraining by language modeling has become popular but we have yet to understand what language models learn during that process. |
| Approach: | They propose diagnostics that ask questions about information used by language models for generating predictions in context. |
| Outcome: | The proposed diagnostics can be used to study the popular BERT model . they show that the model can distinguish good from bad completions, but struggles with inference and role-based event prediction. |
The effects of distance on NPI illusive effects in BERT (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have examined the syntactic capabilities of pre-trained language models, such as BERT. |
| Approach: | They examine the syntactic capabilities of pre-trained language models by using psycholinguistic stimuli. |
| Outcome: | The proposed model is highly sensitive to hierarchical or linear information compared to hierarical layers . |
Understanding by Understanding Not: Modeling Negation in Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Negation is a core construction in natural language, but state-of-the-art pre-trained language models often handle it incorrectly. |
| Approach: | They propose to augment language modeling objective with unlikelihood objective based on negated generic sentences from a raw text corpus. |
| Outcome: | The proposed approach reduces the top 1 error rate to 4% on negated LAMA dataset and improves on negating NLI benchmarks. |
Transformers in the loop: Polarity in neural models of language (2022.acl-long)
Copied to clipboard
| Challenge: | Recent Transformer-based language representation models (LRMs) show impressive results on practical text analysis tasks, but do they have access to complex linguistic notions? |
| Approach: | They propose to use polarity as a case study to compare metrics derived from language models to human judgments obtained in psycholinguistic experiments. |
| Outcome: | The proposed model is more accurate than linguistic theory predictions for polarity, and allows us to use language models to discover new insights into natural language grammar beyond existing linguistic theories. |
How Fast can BERT Learn Simple Natural Language Inference? (2021.eacl-main)
Copied to clipboard
| Challenge: | Efficiency of learning of BERT is very slow due to hidden dataset bias . however, some studies show that it can learn with surface clues/patterns . |
| Approach: | They propose to use a simple entailment judgment case to test whether BERT can learn without hidden dataset bias. |
| Outcome: | The proposed case shows that BERT can learn without hidden bias without utilizing dataset bias. |
Exploring the Role of BERT Token Representations to Explain Sentence Probing Results (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on enhancing existing models with the primary objective of improving downstream performance on various NLP tasks. |
| Approach: | They propose to use BERT to encode meaningful knowledge in token representations to explain probing results. |
| Outcome: | The proposed model can detect syntactic and semantic abnormalities and distinguish between grammatical number and tense subspaces. |
Abstraction not Memory: BERT and the English Article System (2022.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained models are the state of the art in linguistics. |
| Approach: | They compare the performance of pre-trained and native English language models on the task of article prediction set up as a three way choice (a/an, the, zero) they argue that BERT captures a high level generalisation of article use akin to human intuition. |
| Outcome: | The proposed model outperforms humans on the linguistically interesting task of article prediction. |
A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)
Copied to clipboard
| Challenge: | a new study examines the current state of knowledge about the BERT model . the model is a stack of transformer encoder layers that are based on multiple self-attention ''heads'' |
| Approach: | They present a survey of over 150 studies of the popular Transformer-based model BERT . they discuss the current state of knowledge about how BERT works and how it is represented . |
| Outcome: | The proposed model is based on the Transformer-based model with state-of-the-art results . the proposed model has little cognitive motivation and is too small to perform ablation studies . |
On the use of BERT for Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Existing studies on using pretrained language models for supervised NMT have not been successful. |
| Approach: | They propose to integrate BERT pretrained models with supervised NMT models by using monolingual data. |
| Outcome: | The proposed models improve translation quality in English-German, English-Russian and IWSLT14 datasets. |