| Challenge: | BERT is a language representation model that has performed well in diverse language understanding benchmarks. |
| Approach: | They perform experiments to unpack the elements of English language structure learned by BERT. |
| Outcome: | The proposed model outperforms state-of-the-art models in the GLUE benchmark by a significant margin. |
Similar Papers
Finding Structural Knowledge in Multimodal-BERT (2022.acl-long)
Copied to clipboard
| Challenge: | Several multimodal-BERT models learn contextualized embeddings through training on linguistic data and visual data. |
| Approach: | They propose to make the structure of language and visuals explicit by a dependency parse . they also propose to encode the scene tree in the multimodal-BERT models . |
| Outcome: | The proposed models do not encode the scene trees in the language description. |
What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | In addition, information on part-of-speech tagging is spread over different parts of the network and the pipeline might not be as neat as it seems. |
| Approach: | They propose to probe Dutch BERT-based model and multilingual BERT model for Dutch NLP tasks to see if this holds true for other languages. |
| Outcome: | The proposed model is based on a Dutch model and a multilingual model for Dutch NLP tasks. |
Does Chinese BERT Encode Word Structure? (2020.coling-main)
Copied to clipboard
| Challenge: | Existing work has focused on analyzing the features captured by representative models such as BERT . however, little work has investigated word features for character languages such as Chinese . |
| Approach: | They investigate Chinese BERT using attention weight distribution statistics and probing tasks to understand word features. |
| Outcome: | The proposed model improves syntactic, semantic and word sense knowledge on a wide range of NLP tasks. |
Does BERT Rediscover a Classical NLP Pipeline? (2022.coling-1)
Copied to clipboard
| Challenge: | Existing theories of BERT's structure lack conclusive empirical support . however, there is scepticism about the premises of probing itself . |
| Approach: | They propose a new probe called GridLoc that can take into account token positions, training rounds, and random seeds. |
| Outcome: | The proposed probe detects other, stronger regularities suggesting appeals to layer depth may not be the preferable mode of explanation for BERT’s inner workings. |
CxGBERT: BERT meets Construction Grammar (2020.coling-main)
Copied to clipboard
| Challenge: | lexico-semantic elements capture a large amount of linguistic information, but they do not capture all information contained in text. |
| Approach: | They propose to use BERT to train a model that uses a deep bidirectional transformer to capture a significant amount of lexico-semantic information. |
| Outcome: | The proposed model captures lexico-semantic information, but it is redundantly encoded in lexical information. |
Exploring the Role of BERT Token Representations to Explain Sentence Probing Results (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on enhancing existing models with the primary objective of improving downstream performance on various NLP tasks. |
| Approach: | They propose to use BERT to encode meaningful knowledge in token representations to explain probing results. |
| Outcome: | The proposed model can detect syntactic and semantic abnormalities and distinguish between grammatical number and tense subspaces. |
A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)
Copied to clipboard
| Challenge: | a new study examines the current state of knowledge about the BERT model . the model is a stack of transformer encoder layers that are based on multiple self-attention ''heads'' |
| Approach: | They present a survey of over 150 studies of the popular Transformer-based model BERT . they discuss the current state of knowledge about how BERT works and how it is represented . |
| Outcome: | The proposed model is based on the Transformer-based model with state-of-the-art results . the proposed model has little cognitive motivation and is too small to perform ablation studies . |
Let’s Play Mono-Poly: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses (2021.tacl-1)
Copied to clipboard
| Challenge: | Pre-trained language models encode rich information about linguistic structure but their knowledge about lexical polysemy remains unclear. |
| Approach: | They propose a setup for analyzing lexical polysemy knowledge in pre-trained language models and multilingual BERT models by analyzing different sense distributions and controlling for parameters that are highly correlated with polysyntax. |
| Outcome: | The proposed model can be used to analyze lexical polysemy in English, French, Spanish, and Greek and in multilingual BERT. |
Probe-Less Probing of BERT’s Layer-Wise Linguistic Knowledge with Masked Word Prediction (2022.naacl-srw)
Copied to clipboard
| Challenge: | Among studies on localization of linguistic knowledge, it is unclear what information is encoded in each layer. |
| Approach: | They analyze BERT’s layer-wise masked word prediction on an English corpus and find syntactic and semantic information is encoded at different layers for words of different syntaktic categories. |
| Outcome: | The proposed model outperforms state-of-the-art models in many downstream tasks. |
Revealing the Dark Secrets of BERT (D19-1)
Copied to clipboard
| Challenge: | Existing models of BERT-based learning systems are lacking specific mechanisms that contribute to its success. |
| Approach: | They propose to use GLUE tasks to analyze the interpretation of self-attention, which is one of the underlying components of BERT. |
| Outcome: | The proposed model outperforms the regular model on GLUE tasks by disabling attention in certain heads. |