Is a cute puyfred cute? Context-dependent form-meaning systematicity in LLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | valence is encoded in meaningful ways in large language models and in some LLMs, pseudowords affect the representation of whole sentences similarly to words. |
| Approach: | They investigate how LLMs represent valence, a key semantic attribute, and how they deal with contextualisation of pseudowords in sentences. |
| Outcome: | The results show that the models represent valence, a key semantic attribute, in sentences and in context, and that they handle the contextualisation of pseudowords differently. |
Similar Papers
Understanding Subword Compositionality of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) take sequences of subwords as input, requiring them to compose subword representations into meaningful word-level representations. |
| Approach: | They propose to probe how large language models compose subword information . they find structural similarity, semantic decomposability, and form retention are key aspects . |
| Outcome: | The proposed models can be classified into three distinct groups, the authors show . they show that they can achieve great performance when probing layer by layer their sensitivity to semantic decompositionality . |
Unlike “Likely”, “Unlike” is Unlikely: BPE-based Segmentation hurts Morphological Derivations in LLMs (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) use subword vocabularies to process and generate text. |
| Approach: | They find that Large Language Models (LLMs) perform poorly at handling some types of affixations because subwords are marked as initial- or intra-word . |
| Outcome: | The largest models trained on enough data can mitigate this tendency because initial- and intra-word embeddings are aligned; in-context learning also helps when all examples are selected in a consistent way; but only morphological segmentation can achieve a near-perfect accuracy. |
Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models that represent different senses of words in context are not accurate for polysemous words. |
| Approach: | They propose a multilingual dataset that evaluates the ability of models to accurately represent different lexical-semantic relations such as homonymy and synonymy. |
| Outcome: | The proposed models can disambiguate homonyms in context, but fail to represent words with different senses when occurring in similar sentences. |
Patterns of Polysemy and Homonymy in Contextualised Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study has focused on homonymy, a variety of multiplicity of meanings exemplified by word forms with unrelated meanings. |
| Approach: | They investigate the extent to which contextualised embeddings reflect traditional distinctions of polysemy and homonymy. |
| Outcome: | The proposed model shows that it can distinguish between polysemy and homonymy . it shows that the model fails to replicate the results of the human-annotated dataset . |
Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing research suggests that contextual representations of large language models exhibit subpar performance in downstream tasks, struggling to fully capture the semantic nuances of words. |
| Approach: | They investigate the bottom-up evolution of lexical semantics for a popular LLM . they probing its hidden states at the end of each layer using a contextualized word identification task . |
| Outcome: | The proposed model is able to encode lexical semantics in lower layers while achieving weaker induction in higher layers. |
Exploring Layer-wise Representations of English and Chinese Homonymy in Pre-trained Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | lexical ambiguity can arise due to the misunderstanding of its multiple senses. |
| Approach: | They propose to use part of speech to examine homonyms in Chinese and English . they find no universal layer depth excels in differentiating homnomial representations . |
| Outcome: | The proposed model improves contextualization of homonym representations in Chinese . the results challenge the simplistic understanding of their inner workings, the authors say . |
Deep Generative Model for Joint Alignment and Word Representation (N18-1)
Copied to clipboard
| Challenge: | EmbedAlign model embeds words in their complete observed context and learns by marginalisation of latent lexical alignments. |
| Approach: | They exploit translation as a distributional context and embed words as posterior probability densities, rather than point estimates, which allows them to compare words in context using a measure of overlap between distributions. |
| Outcome: | The proposed model performs on a range of lexical semantics tasks and achieves competitive results on benchmarks including natural language inference, paraphrasing, and text similarity. |
CUTE: Measuring LLMs’ Understanding of Their Tokens (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) perform well on a wide variety of tasks, authors say . they lack direct access to characters, which can be difficult to generalize to new languages . |
| Approach: | They propose a benchmark to test the orthographic knowledge of Large Language Models . they find that most LLMs seem to know the spelling of their tokens - yet fail to manipulate text . |
| Outcome: | The proposed benchmark tests the orthographic knowledge of large language models . it finds that most LLMs seem to know the spelling of their tokens, but fail to manipulate text . |
Putting Words in Context: LSTM Language Models and Lexical Ambiguity (P19-1)
Copied to clipboard
| Challenge: | In language, a word can contribute a very different meaning depending on the context . lexical ambiguity involves both morphosyntactic and semantic aspects . |
| Approach: | They propose a method to probe hidden representations for lexical and contextual information about words. |
| Outcome: | The proposed method shows that both types of information are represented to a large extent, but there is room for improvement for contextual information. |
LLM See, LLM Do: Leveraging Active Inheritance to Target Non-Differentiable Objectives (2024.emnlp-main)
Copied to clipboard
| Challenge: | Historically, high-quality labeled data has been costly to curate due to scarcity of available data and financial cost. |
| Approach: | They characterize the impact of passive inheritance of model properties by studying how the source of synthetic data shapes models’ internal biases, calibration and preferences, and their generations’ textual attributes. |
| Outcome: | The proposed model inheritance can increase lexical diversity or reduce toxicity. |