On the Interplay Between Fine-tuning and Composition in Transformers (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained transformer language models have shown remarkable performance on a variety of NLP tasks. |
| Approach: | They propose to fine-tune transformer language models on a paraphrase and sentiment task and analyze their results to determine whether they benefit compositionality. |
| Outcome: | The proposed model performance on a paraphrase and sentiment task is compared with pre-trained models on lexical-level representations. |
Similar Papers
On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | linguistic knowledge encoded in pre-trained contextual embeddings is poorly understood . fine-tuning can be used to investigate the representations of pre-train models . |
| Approach: | They propose to investigate fine-tuning of contextualized embedding models through sentence-level probing. |
| Outcome: | The proposed method improves probing accuracy for three pre-trained models. |
Assessing Phrasal Representation and Composition in Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for deep transformers are able to combine word meanings into phrase meanings, but they lack a clear understanding of how they handle complex linguistic inputs. |
| Approach: | They propose to analyze phrasal representations in pre-trained transformers to determine whether they reflect sophisticated composition of phrase meaning. |
| Outcome: | The proposed models are able to combine meaning units into larger units, a phenomenon known as composition, and reflects human understanding of meaning. |
LexFit: Lexical Fine-Tuning of Pretrained Language Models (2021.acl-long)
Copied to clipboard
| Challenge: | Transformer-based language models implicitly store a wealth of lexical semantic knowledge, but it is non-trivial to extract that knowledge effectively from their parameters. |
| Approach: | They propose to expose and enrich lexical knowledge from transformer-based language models to serve as effective decontextualized word encoders even when fed input words "in isolation" |
| Outcome: | The proposed model outperforms standard static WEs and vanilla LMs in lexical tasks over four established tasks in 8 languages. |
Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Backchannels and fillers are important linguistic expressions in dialogue, but often ignored in modern transformer-based language models. |
| Approach: | They use clustering analysis to learn backchannels and fillers in dialogues in English and Japanese and use natural language generation metrics to confirm this. |
| Outcome: | The proposed models can learn representations of backchannels and fillers using three fine-tuning strategies. |
Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on how to encode compositional task structure has been limited by semantic parsing and multihop reasoning for the purpose of Q&A. |
| Approach: | They propose an approach to decomposing a target task into component tasks and fine-tuning smaller LMs on a curriculum of such component tasks. |
| Outcome: | The proposed approach outperforms end-to-end learning even with equal data, and gets better as more component tasks are modeled. |
How do Transformer Embeddings Represent Compositions? A Functional Analysis (2025.findings-acl)
Copied to clipboard
| Challenge: | Despite the popularity of transformer-based models, little is known about how they represent compound words and whether they are compositional. |
| Approach: | They evaluate compositionality in mistral, OpenAI Large, and Google embedding models and compare them with BERT. |
| Outcome: | The proposed models perform best in addition, multiplication, dilation, regression, and the classic vector addition model performs almost as well as any other model. |
The Impact of Depth on Compositional Generalization in Transformer Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | In this paper, we test the hypothesis that deeper transformers generalize more compositionally. |
| Approach: | They propose to add layers to transformers to generalize more compositionally . they propose to fine-tune the models so that the total number of parameters is constant . |
| Outcome: | The proposed model generalizes more compositionally than shallower models, but returns diminish . the proposed model can be made shallower without sacrificing performance . |
How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models are ineffective, despite their potential, pre-training models suffer from important weaknesses. |
| Approach: | They analyze the extent to which the isotropy of the embedding space changes after fine-tuning. |
| Outcome: | The proposed model improves the isotropy of embedding space after fine-tuning . the model can encode linguistic properties, but lacks the social bias needed to improve it . |
What Context Features Can Transformer Language Models Use? (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. |
| Approach: | They propose to use lexical and structural information to ablate usable information in transformer language models. |
| Outcome: | The proposed model improves when conditioning on contexts of thousands of previous tokens. |
Quantifying Compositionality of Classic and State-of-the-Art Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Static word embeddings make strong claims about compositionality, but the SOTA generative models go too far in the other direction. |
| Approach: | a new study evaluates the compositionality of word embeddings by canonical correlation analysis . strong compositional signals are observed in later training stages across data modalities . |
| Outcome: | a new evaluation of compositional models shows that they exploit access meanings when justified . strong compositional signals are observed in later training stages and in deeper layers of the transformer-based model before a decline at the top layer. |