Quantifying Compositionality of Classic and State-of-the-Art Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Static word embeddings make strong claims about compositionality, but the SOTA generative models go too far in the other direction. |
| Approach: | a new study evaluates the compositionality of word embeddings by canonical correlation analysis . strong compositional signals are observed in later training stages across data modalities . |
| Outcome: | a new evaluation of compositional models shows that they exploit access meanings when justified . strong compositional signals are observed in later training stages and in deeper layers of the transformer-based model before a decline at the top layer. |
Similar Papers
How do Transformer Embeddings Represent Compositions? A Functional Analysis (2025.findings-acl)
Copied to clipboard
| Challenge: | Despite the popularity of transformer-based models, little is known about how they represent compound words and whether they are compositional. |
| Approach: | They evaluate compositionality in mistral, OpenAI Large, and Google embedding models and compare them with BERT. |
| Outcome: | The proposed models perform best in addition, multiplication, dilation, regression, and the classic vector addition model performs almost as well as any other model. |
Additive Compositionality of Word Vectors (D19-55)
Copied to clipboard
| Challenge: | Existing research on justifying additive compositionality of word embedding models requires a rather strong assumption of uniform word distribution. |
| Approach: | They propose to relax the assumption of uniform word distribution and propose more realistic conditions for proving additive compositionality. |
| Outcome: | The proposed model improves on word similarity and noisy sentence similarity. |
Simple and effective data augmentation for compositional generalization (2024.naacl-long)
Copied to clipboard
| Challenge: | Compositional generalization is the ability of a system to correctly predict the meaning of complex sentences when trained on simpler sentences. |
| Approach: | They propose to use data augmentation methods to generate additional training data by sampling from an augmentation distribution to generalize to the out-of-distribution test data. |
| Outcome: | The proposed method outperforms existing methods that sampled from the training distribution and outperformed existing methods. |
What’s in Your Embedding, And How It Predicts Task Performance (C18-1)
Copied to clipboard
| Challenge: | Attempts to find a single technique for general-purpose intrinsic evaluation of word embeddings have so far not been successful. |
| Approach: | They propose a method that quantifies interpretable characteristics of word vector neighborhoods and shows how they correlate with performance on 14 extrinsic and intrinsic task datasets. |
| Outcome: | The proposed approach enables multi-faceted evaluation, parameter search, and generally – a more principled, hypothesis-driven approach to development of distributional semantic representations. |
Montague semantics and modifier consistency measurement in neural language models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies on distributional language models have been focused on linguistics and their relationship with semantic formalisms for decades. |
| Approach: | They propose a method for measuring compositional behavior in contemporary language embedding models by introducing three new tests inspired by Montague semantics. |
| Outcome: | The proposed method measures compositional behavior in language embedding models on adjectival modifier phenomena in adjective-noun phrases. |
Combine to Describe: Evaluating Compositional Generalization in Image Captioning (2022.acl-srw)
Copied to clipboard
| Challenge: | Recent work on compositionality has focused on the ability to combine simpler concepts to understand & generate arbitrarily more complex conceptual structures. |
| Approach: | They propose to use a set of image captioning models to benchmark their compositional generalization properties. |
| Outcome: | The proposed models do not generalize in terms of systematicity and productivity, but are robust to synonym substitutions. |
On Evaluating Multilingual Compositional Generalization with Translated Datasets (2023.acl-long)
Copied to clipboard
| Challenge: | a growing amount of research investigating compositional generalization in NLP is done on English . a critical semantic distortion is a limitation of the translation of datasets . |
| Approach: | They propose to translate a dataset for evaluating compositional generalization in semantic parsing. |
| Outcome: | The proposed benchmarks show that the translation of the MCWQ dataset suffers from semantic distortion. |
Building Static Embeddings from Contextual Ones: Is It Useful for Building Distributional Thesauri? (2022.lrec-1)
Copied to clipboard
| Challenge: | contextual language models are dominant in the field of Natural Language Processing, but they are not suitable for all uses. |
| Approach: | They propose a method for building word or type-level embeddings from contextual models . they evaluate a large set of English nouns from the perspective of extracting semantic similarity relations . |
| Outcome: | The proposed method can be used to build word or type embeddings from contextual models . it can be exploited for a wide set of English nouns, showing it can improve distributional thesauri . |
Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for vision-language models treat compositionality and long-caption understanding in isolation. |
| Approach: | They analyze when compositional reasoning and long-caption understanding transfer across tasks and when this relationship fails. |
| Outcome: | The proposed model can generalize on poorly grounded captions and with strong visual grounding, while architectural choices can limit compositional learning. |
Understanding the Influence of Synthetic Data for Text Embedders (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in general purpose text embedders have been driven by training on synthetic training data. |
| Approach: | They propose to use GPT-4 to produce high quality synthetic data that expands existing training datasets for embeddings to new tasks. |
| Outcome: | The proposed dataset is high quality and leads to consistent improvements in performance. |