| Challenge: | linguistic theories have argued that language models have largely achieved grammatical competence, but they will assign non-zero probability to all strings. |
| Approach: | They propose a theoretical framework for analyzing string probabilities in linguistics based on simple assumptions about the generative process of corpus data. |
| Outcome: | The proposed framework makes three predictions using 280K sentence pairs in English and Chinese. |
Similar Papers
Implicit Representations of Grammaticality in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Pretrained language models generate grammatically well-formed text and discriminate well between grammatical and ungrammatically sentences in tightly controlled minimal pairs. |
| Approach: | They propose a method to train pretrained LMs for representations of grammaticality by applying perturbations to a naturalistic text corpus. |
| Outcome: | The proposed model outperforms probability-based models on human-curated grammaticality judgment benchmarks and performs worse than string probabilities on plausibility benchmarks. |
How to Compute the Probability of a Word (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language models estimate a probability distribution over strings in a natural language . many recent linguistic studies have been incorrectly computing word probabilities . |
| Approach: | They propose to use the correct method to compute word probabilities . they highlight issues when relying on models that use end-of-word tokenisers . |
| Outcome: | Empirically, correcting the widespread bug affects measured outcomes in sentences and lexical optimisation analyses. |
Constructions are Revealed in Word Distributions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Construction grammar posits that constructions are form-meaning pairings that are acquired through experience with language. |
| Approach: | They propose to use a RoBERTa model to model constructions as patterns of statistical affinity . they show that statistical affinity is likely an important, but partial, signal available to learners . |
| Outcome: | The proposed model shows that constructions will be revealed as patterns of statistical affinity . the proposed model is based on a model that is able to distinguish constructions from text . |
Function Words as Statistical Cues for Language Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies have argued that function words aid learning abstract grammatical knowledge from linear input. |
| Approach: | They examine the statistical distribution of function words and their properties . they show that function words are reliable, diverse, and informative . |
| Outcome: | The results show that function words preserve high frequency, reliable syntactic association, phrase-boundary alignment and are informative to structural dependency. |
How to Make the Most of LLMs’ Grammatical Knowledge for Acceptability Judgments (2025.naacl-long)
Copied to clipboard
Yusuke Ide, Yuto Nishida, Justin Vasselli, Miyu Oba, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe
| Challenge: | Conventional approaches compare sentence probabilities directly, but large language models (LLMs) provide nuanced evaluation methods using prompts and templates. |
| Approach: | They propose to derive acceptability judgments from large language models using prompts and templates to comprehensively evaluate their grammatical knowledge. |
| Outcome: | The proposed methods excel in different linguistic phenomena, suggesting they access different aspects of the LLMs’ grammatical knowledge. |
Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work has shown that language models can select the most likely or plausible of a set of possible events, but they are far from robust. |
| Approach: | They focus on whether language models can select the most likely or plausible of a set of possibilities and compare them to a broader behavior that humans exhibit largely unconsciously. |
| Outcome: | The proposed models perform worse than expected under certain conditions, compared with Llama 3, Gemma 2, and Mistral NeMo, and they are significantly more sensible than leaves. |
A Construction Grammar Corpus of Varying Schematicity: A Dataset for the Evaluation of Abstractions in Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been developed without a theoretical framework . evaluating and improving LLMs will benefit from theoretical frameworks that enable comparison of structures of human language and model of language built up by LLM. |
| Approach: | They propose to use a construction grammar schema corpus to compare human grammar to LLMs' model of language. |
| Outcome: | The proposed corpus shows that even the largest LLMs are limited to more substantive constructions and do not recognize similarity of purely schematic constructions. |
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent studies have shown that language models capture different types of knowledge regarding facts or commonsense knowledge. |
| Approach: | They examine how language models can be calibrated to make their confidence scores correlate better with the likelihood of correctness. |
| Outcome: | The proposed calibration methods improve confidence scores on QA tasks and improve accuracy. |
Can Language Models Learn Typologically Implausible Languages? (2026.tacl-1)
Copied to clipboard
| Challenge: | Language models provide a naturalistic framework for studying artificial language learning . authors: typological universals and tendencies are thought to be caused by a learning bias . |
| Approach: | They propose to train LMs on highly naturalistic counterfactual versions of English and Japanese . they show that LM learn subtly implausible languages more slowly . |
| Outcome: | The proposed language models learn subtly implausible languages more slowly compared to human models . the findings suggest that LMs exhibit typologically aligned learning preferences . |
Do Language Models Exhibit Human-like Structural Priming Effects? (2024.findings-acl)
Copied to clipboard
| Challenge: | a recent exposure to a structure facilitates processing of the same structure, a study finds . structural priming is well attested in humans, for both language production and comprehension . |
| Approach: | They use the structural priming paradigm to investigate where priming effects manifest . they find that rarer elements within a prime increase priming effect . |
| Outcome: | The findings provide an important piece in the puzzle of understanding how properties within their context affect structural prediction in language models. |