Syntax-guided Neural Module Distillation to Probe Compositionality in Sentence Embeddings (2023.eacl-main)
Copied to clipboard
| Challenge: | Past work on sentence embedding models faces issues determining the causal impact of implicit syntax representations. |
| Approach: | They construct a neural module net based on a transformer model and train it end-to-end to approximate the sentence’s embedding. |
| Outcome: | The proposed model captures whether syntax is a strong model of its compositional ability. |
Similar Papers
Quantifying Compositionality of Classic and State-of-the-Art Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Static word embeddings make strong claims about compositionality, but the SOTA generative models go too far in the other direction. |
| Approach: | a new study evaluates the compositionality of word embeddings by canonical correlation analysis . strong compositional signals are observed in later training stages across data modalities . |
| Outcome: | a new evaluation of compositional models shows that they exploit access meanings when justified . strong compositional signals are observed in later training stages and in deeper layers of the transformer-based model before a decline at the top layer. |
Assessing Composition in Sentence Vector Representations (C18-1)
Copied to clipboard
| Challenge: | opacity of sentence vector representations is a challenge to achieving language understanding . current neural network models are unable to capture meaning information in dense vectors . |
| Approach: | They propose a method that targets compositional meaning information in sentence embeddings with a high degree of precision and control. |
| Outcome: | The proposed method extracts useful information about the different capacities of existing sentences models. |
How do Transformer Embeddings Represent Compositions? A Functional Analysis (2025.findings-acl)
Copied to clipboard
| Challenge: | Despite the popularity of transformer-based models, little is known about how they represent compound words and whether they are compositional. |
| Approach: | They evaluate compositionality in mistral, OpenAI Large, and Google embedding models and compare them with BERT. |
| Outcome: | The proposed models perform best in addition, multiplication, dilation, regression, and the classic vector addition model performs almost as well as any other model. |
LexSym: Compositionality as Lexical Symmetry (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to generalize compositional models fail to generalise from small datasets. |
| Approach: | They propose a domain-general and model-agnostic formulation of compositionality as a constraint on symmetries of data distributions rather than models. |
| Outcome: | The proposed procedure matches or surpasses state-of-the-art, task-specific models on COGS semantic parsing, SCAN and Alchemy instruction following, and CLEVR-CoGenT visual question answering datasets. |
The Paradox of the Compositionality of Natural Language: A Neural Machine Translation Case Study (2022.acl-long)
Copied to clipboard
| Challenge: | Obtaining human-like performance in NLP is often argued to require compositional generalisation. |
| Approach: | They re-instantiate three compositionality tests from the literature and reformulate them for neural machine translation. |
| Outcome: | The proposed models are more compositional than models trained on more data, the authors show . they also show that some non-compositional behaviours are mistakes, whereas others reflect natural variation in data. |
Multi-Step Inference for Reasoning Over Paragraphs (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for complex reasoning use symbols or black-box transformers . a compositional model can chain together free-form predicates and logical connectives . |
| Approach: | They propose a compositional model that finds relevant sentences and then chains them together using neural modules. |
| Outcome: | The proposed model improves performance on a recently-introduced dataset. |
Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learn sentence embeddings do not capture the semantic similarity of sentences. |
| Approach: | They propose a framework that integrates compositional sentence operations into the embedding space and optimizes operator networks and a bottleneck encoder-decoder model to produce meaningful and interpretable sentence embeddables. |
| Outcome: | The proposed framework improves the interpretability of sentence embeddings on four textual generation tasks while maintaining strong performance on traditional semantic similarity tasks. |
Montague semantics and modifier consistency measurement in neural language models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies on distributional language models have been focused on linguistics and their relationship with semantic formalisms for decades. |
| Approach: | They propose a method for measuring compositional behavior in contemporary language embedding models by introducing three new tests inspired by Montague semantics. |
| Outcome: | The proposed method measures compositional behavior in language embedding models on adjectival modifier phenomena in adjective-noun phrases. |
Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in mechanistic interpretability have made progress in identifying circuits, the minimal computational subgraphs responsible for a model’s behavior on specific tasks. |
| Approach: | They propose to analyze circuits for highly compositional subtasks within a transformer-based language model to determine their modularity and how they relate to each other. |
| Outcome: | The proposed approach shows that the circuits identified exhibit notable node overlap and cross-task faithfulness. |
Measuring Idiomaticity in Text Embedding Models with epsilon-compositionality (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing studies on compositionality of text embedding models have limited understanding of the principle . idioms have traditionally been seen as non-compositional . |
| Approach: | They propose to use formal definitions to define compositionality in text embedding models . they find that most models differentiate between idiomatic and non-idiomatic phrases . |
| Outcome: | The proposed model is able to differentiate between idiomatic and non-idiomatic phrases, the authors show . |