On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | linguistic knowledge encoded in pre-trained contextual embeddings is poorly understood . fine-tuning can be used to investigate the representations of pre-train models . |
| Approach: | They propose to investigate fine-tuning of contextualized embedding models through sentence-level probing. |
| Outcome: | The proposed method improves probing accuracy for three pre-trained models. |
Similar Papers
A Closer Look at How Fine-tuning Changes BERT (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained contextualized representations are used to analyze information in NLP . however, how fine-tuning changes the underlying embedding space is less studied . |
| Approach: | They propose to use probing techniques to analyze how fine-tuning changes the embedding space of pre-trained contextualized representations. |
| Outcome: | The proposed model improves classification performance by increasing the distances between examples associated with different labels. |
Can Edge Probing Tests Reveal Linguistic Knowledge in QA Models? (2022.coling-1)
Copied to clipboard
| Challenge: | grammatical knowledge is encoded in large pre-trained language models (LMs) this is done through supervised classification tasks to predict the grammamatical properties of a span using only the token representations coming from the LM encoder. |
| Approach: | They propose to use a supervised 'edge probing' task to detect grammatical knowledge in large pre-trained language models (LMs) this is done by encoding grammamatical properties using only token representations coming from the LM encoder. |
| Outcome: | The proposed model performs well when fine-tuned or in adversarial situations where the model is forced to learn wrong correlations. |
On the Interplay Between Fine-tuning and Composition in Transformers (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained transformer language models have shown remarkable performance on a variety of NLP tasks. |
| Approach: | They propose to fine-tune transformer language models on a paraphrase and sentiment task and analyze their results to determine whether they benefit compositionality. |
| Outcome: | The proposed model performance on a paraphrase and sentiment task is compared with pre-trained models on lexical-level representations. |
Predicting Fine-Tuning Performance with Probing (2022.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale neural models have recently demonstrated impressive performance in language understanding tasks, typically evaluated by their fine-tuned performance. |
| Approach: | They propose to use probing to extract a proxy signal widely used in model development to predict fine-tuning performance. |
| Outcome: | The proposed method predicts fine-tuning performance with errors 40% - 80% smaller than baselines. |
On the Nature of BERT: Correlating Fine-Tuning and Linguistic Competence (2022.coling-1)
Copied to clipboard
| Challenge: | Several studies on the interpretation of Neural Language Models (NLMs) focus on the linguistic generalization abilities of pre-trained models, but little attention is paid to how the linguistic knowledge of the models changes during fine-tuning. |
| Approach: | They propose to examine whether a wide range of linguistic phenomena are forgotten during fine-tuning and whether it is possible to predict the fine- tuned accuracy solely relying on the assessed linguistic competence. |
| Outcome: | The proposed model can predict the evolution of written language competence of native language learners based on the assessed linguistic competence. |
How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models are ineffective, despite their potential, pre-training models suffer from important weaknesses. |
| Approach: | They analyze the extent to which the isotropy of the embedding space changes after fine-tuning. |
| Outcome: | The proposed model improves the isotropy of embedding space after fine-tuning . the model can encode linguistic properties, but lacks the social bias needed to improve it . |
On the Importance of Data Size in Probing Fine-tuned Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Several studies have investigated the reasons behind the effectiveness of fine-tuning, usually through the lens of probing. |
| Approach: | They propose to investigate the reasons behind the effectiveness of fine-tuning by examining the impact of data size on the extent of encoded linguistic knowledge. |
| Outcome: | The proposed probes show that the size of the training data affects the recoverability of the changes made to the model’s linguistic knowledge. |
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance? (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural models have established state-of-the-art performance on several NLP benchmarks, but little is understood about the mechanisms by which they operate. |
| Approach: | They examine the probing paradigm through a set of controlled synthetic tasks and show that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself. |
| Outcome: | The proposed model can encode linguistic properties above chance-level even when distributed in the data as random noise, reversing the interpretation of absolute claims on probing tasks. |
On the Transformation of Latent Space in Fine-Tuned NLP Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | a large body of work analyzed the knowledge learned within representations of pre-trained models. |
| Approach: | They use hierarchical clustering to discover latent concepts in representational space . they compare pre-trained and fine-tuned models and perform a thorough analysis . |
| Outcome: | The results show that the model space evolves towards task-specific concepts whereas the lower layers retain generic concepts acquired in the pre-trained model. |
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models can be fine-tuned to produce state-of-the-art results for a wide range of language understanding tasks. |
| Approach: | They propose to analyze fine-tuning through the lens of intrinsic dimension . they show that pre-trained models have a low intrinsic dimension reparameterization . |
| Outcome: | The proposed model can achieve 90% of the full parameter performance levels on MRPC with low data regime. |