Using Perspectival Words Is Harder Than Vocabulary Words for Humans —and Even More So for Multimodal Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations of multimodal language models focus on vocabulary words with relatively stable, context-independent meanings in conversation, such as object names, colors, and verbs. |
| Approach: | They compare human and multimodal language models in their use of three word types: vocabulary, possessives, and demonstratives. |
| Outcome: | The models approach human-level performance on using vocabulary, but exhibit clear deficits with possessives and even greater difficulties with demonstratives. |
Similar Papers
Naming, Describing, and Quantifying Visual Objects in Humans and LLMs (2024.acl-short)
Copied to clipboard
| Challenge: | Recent work has highlighted that speakers display a wide range of variability when asked to utter sentences, resulting in inter-speaker variability but also variability over time for the same speaker. |
| Approach: | They evaluate Vision & Language Large Language Models (VLLMs) on three categories where humans show great subjective variability concerning the distribution over plausible labels. |
| Outcome: | The proposed models can mimic human distributions over plausible labels, but fail to assign quantifiers, a task that requires more accurate, high-level reasoning. |
Comparing human and language models sentence processing difficulties on complex structures (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) that converse with humans are a reality, but do LLMs experience human-like processing difficulties? |
| Approach: | They systematically compare human and LLM sentence comprehension across seven challenging linguistic structures. |
| Outcome: | The proposed model achieves near perfect accuracy on non-GP structures, but struggles on GP structures. |
Unlike “Likely”, “Unlike” is Unlikely: BPE-based Segmentation hurts Morphological Derivations in LLMs (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) use subword vocabularies to process and generate text. |
| Approach: | They find that Large Language Models (LLMs) perform poorly at handling some types of affixations because subwords are marked as initial- or intra-word . |
| Outcome: | The largest models trained on enough data can mitigate this tendency because initial- and intra-word embeddings are aligned; in-context learning also helps when all examples are selected in a consistent way; but only morphological segmentation can achieve a near-perfect accuracy. |
Are Multimodal Large Language Models Pragmatically Competent Listeners in Simple Reference Resolution Tasks? (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models are unable to resolve references to abstract visual stimuli, such as color patches and color grids, but their pragmatic capabilities are still a challenge for state-of-the-art MLLMs. |
| Approach: | They investigate whether multimodal large language models are able to resolve references to abstract visual stimuli, such as color patches and color grids, in a well-known reference resolution paradigm. |
| Outcome: | The proposed model can resolve references to abstract visual stimuli in dyadic reference games. |
The Sensitivity of Language Models and Humans to Winograd Schema Perturbations (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are driving recent improvements in perfromance on the Winograd Schema Challenge . a diagnostic dataset shows that these models are sensitive to linguistic perturbations that minimally affect human understanding . |
| Approach: | They propose to use a dataset to test pre-trained language models for the Winograd Schema Challenge . they show that these models are sensitive to linguistic perturbations that minimally affect human understanding . |
| Outcome: | The proposed models are sensitive to linguistic perturbations that minimally affect human understanding. |
Do LLMs Capture Embodied Cognition and Cultural Variation? Cross-Linguistic Evidence from Demonstratives (2026.acl-long)
Copied to clipboard
| Challenge: | a new study examines whether large language models acquire embodied cognition and cultural conventions from training data . demonstratives are a natural lens for evaluating linguistic phenomena that reflect cultural variation . aaron e. duan and j. nà: "the complexity of the language model is a major challenge for LLMs" |
| Approach: | They introduce demonstratives as a probe for grounded knowledge by analyzing 6,400 responses from 320 native speakers. |
| Outcome: | The proposed model fails to understand proximal–distal contrast and shows no cultural differences . the proposed model is a new probe for evaluating embodied cognition and cultural conventions . |
A Targeted Assessment of Incremental Processing in Neural Language Models and Humans (2021.acl-long)
Copied to clipboard
| Challenge: | Using by-word reaction time data, we compare incremental processing in humans and neural language models across a range of structural phenomena. |
| Approach: | They propose to scale up incremental processing in humans and language models by collecting by-word reaction time data for 16 different syntactic test suites. |
| Outcome: | The proposed model outputs match human and model accuracy scores, but underpredict the difference in magnitude of incremental processing difficulty between grammatical and ungrammatically-spaced sentences. |
Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs? (2025.naacl-long)
Copied to clipboard
| Challenge: | Among various types of ambiguity, this study focuses on syntactic ambiguities, specifically relative 1 Dataset available at https://github.com/PortNLP/ MultiWHO. |
| Approach: | They propose to use a dataset to fine-grained evaluate relative clause attachment preferences in ambiguous and unambiguous contexts. |
| Outcome: | The proposed dataset shows that large language models perform well in unambiguous cases, but lack flexibility in human language processing. |
Visual Grounding Helps Learn Word Meanings in Low-Data Regimes (2024.naacl-long)
Copied to clipboard
| Challenge: | Modern neural language models (LMs) require distinctly un-human-like ways to achieve these results. |
| Approach: | They train a diverse set of LM architectures with and without auxiliary visual supervision on datasets of varying scales. |
| Outcome: | The proposed models exhibit better learning of syntactic categories, lexical relations, semantic features, word similarity and alignment with human neural representations. |
Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval? (2026.findings-acl)
Copied to clipboard
| Challenge: | Rapid advances in multimodal large language models have revolutionized cross-modality understanding. |
| Approach: | They propose a method that uses whitening transformations to adjust MLLM representation spaces . they propose ML models that are dominated by textual semantics and visual semantics . |
| Outcome: | The proposed approach improves zero-shot multimodal retrieval performance without fine-tuning efforts. |