| Challenge: | Pre-trained text encoders have advanced the state of the art on many NLP tasks . Qualitative analysis reveals that the model can and often does adjust this pipeline dynamically . |
| Approach: | They aim to quantify where linguistic information is captured within a network model . they aim to use pre-trained text encoders to displace static word embeddings . |
| Outcome: | The proposed model can adjust the pipeline dynamically, revealing lower-level decisions on the basis of disambiguation from higher-level representations. |
Similar Papers
Does BERT Rediscover a Classical NLP Pipeline? (2022.coling-1)
Copied to clipboard
| Challenge: | Existing theories of BERT's structure lack conclusive empirical support . however, there is scepticism about the premises of probing itself . |
| Approach: | They propose a new probe called GridLoc that can take into account token positions, training rounds, and random seeds. |
| Outcome: | The proposed probe detects other, stronger regularities suggesting appeals to layer depth may not be the preferable mode of explanation for BERT’s inner workings. |
What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | In addition, information on part-of-speech tagging is spread over different parts of the network and the pipeline might not be as neat as it seems. |
| Approach: | They propose to probe Dutch BERT-based model and multilingual BERT model for Dutch NLP tasks to see if this holds true for other languages. |
| Outcome: | The proposed model is based on a Dutch model and a multilingual model for Dutch NLP tasks. |
A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)
Copied to clipboard
| Challenge: | a new study examines the current state of knowledge about the BERT model . the model is a stack of transformer encoder layers that are based on multiple self-attention ''heads'' |
| Approach: | They present a survey of over 150 studies of the popular Transformer-based model BERT . they discuss the current state of knowledge about how BERT works and how it is represented . |
| Outcome: | The proposed model is based on the Transformer-based model with state-of-the-art results . the proposed model has little cognitive motivation and is too small to perform ablation studies . |
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Pretraining by language modeling has become popular but we have yet to understand what language models learn during that process. |
| Approach: | They propose diagnostics that ask questions about information used by language models for generating predictions in context. |
| Outcome: | The proposed diagnostics can be used to study the popular BERT model . they show that the model can distinguish good from bad completions, but struggles with inference and role-based event prediction. |
Which *BERT? A Survey Organizing Contextualized Encoders (2020.emnlp-main)
Copied to clipboard
| Challenge: | a survey on language representation learning aims to highlight common themes . we focus on the areas of progress, compared to other fields, and discuss how each area is evaluated. |
| Approach: | They present a survey on language representation learning to highlight common themes . they compare contributions in contextualized text encoders to ideas from other fields . |
| Outcome: | The proposed survey aims to highlight common themes in the field of language representation learning. |
From BERT‘s Point of View: Revealing the Prevailing Contextual Differences (2022.findings-acl)
Copied to clipboard
| Challenge: | BERTology is a new approach to understanding the inner workings of large pretraining language models. |
| Approach: | They propose to invert the probing design to analyze the prevailing differences and clusters in BERT’s high dimensional space by extracting coarse features from masked token representations and predicting them by probing models with access to only partial information. |
| Outcome: | The proposed method extracts coarse features from masked token representations and predicts them by probing models with access to only partial information. |
Visualizing and Understanding the Effectiveness of BERT (D19-1)
Copied to clipboard
| Challenge: | Language model pre-training, such as BERT, has achieved strong performance in many NLP tasks. |
| Approach: | They propose to visualize loss landscapes and optimization trajectories of fine-tuning BERT on specific datasets. |
| Outcome: | The proposed model improves performance and generalization capability across tasks. |
What Does BERT Learn about the Structure of Language? (P19-1)
Copied to clipboard
| Challenge: | BERT is a language representation model that has performed well in diverse language understanding benchmarks. |
| Approach: | They perform experiments to unpack the elements of English language structure learned by BERT. |
| Outcome: | The proposed model outperforms state-of-the-art models in the GLUE benchmark by a significant margin. |
First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT (2021.eacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have demonstrated remarkable zero-shot cross-lingual transfer capabilities. |
| Approach: | They propose to use a layer ablation technique to create a multilingual model that is viewed as a stacking of two sub-networks: a language-agnostic encoder and a task-specific predictor. |
| Outcome: | The proposed model can perform zero-shot cross-lingual transfer for many languages. |
BERTwich: Extending BERT’s Capabilities to Model Dialectal and Noisy Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models like BERT deteriorate in the face of dialect variation or noise. |
| Approach: | They propose to sandwich BERT's encoder stack between additional encoder layers trained to perform masked language modeling on noisy text. |
| Outcome: | The proposed approach promotes zero-shot transfer to dialectal text and reduces embedding space between words and noisy counterparts. |