A Construction Grammar Corpus of Varying Schematicity: A Dataset for the Evaluation of Abstractions in Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been developed without a theoretical framework . evaluating and improving LLMs will benefit from theoretical frameworks that enable comparison of structures of human language and model of language built up by LLM. |
| Approach: | They propose to use a construction grammar schema corpus to compare human grammar to LLMs' model of language. |
| Outcome: | The proposed corpus shows that even the largest LLMs are limited to more substantive constructions and do not recognize similarity of purely schematic constructions. |
Similar Papers
Constructions are Revealed in Word Distributions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Construction grammar posits that constructions are form-meaning pairings that are acquired through experience with language. |
| Approach: | They propose to use a RoBERTa model to model constructions as patterns of statistical affinity . they show that statistical affinity is likely an important, but partial, signal available to learners . |
| Outcome: | The proposed model shows that constructions will be revealed as patterns of statistical affinity . the proposed model is based on a model that is able to distinguish constructions from text . |
CxLM: A Construction and Context-aware Language Model (2022.lrec-1)
Copied to clipboard
| Challenge: | Constructions are direct form-meaning pairs with possible schematic slots . however, these slots are constrained by the embedded construction and the context . we propose that a conditional probability distribution could be described but language models cannot capture this distribution. |
| Approach: | They propose that a conditional probability distribution could describe constructions’ schematic slots. |
| Outcome: | The proposed model predicts masked slots more accurately than baselines and generates structurally and semantically plausible word samples. |
From Parameters to Performance: A Data-Driven Study on LLM Structure and Development (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have revolutionized a wide range of domains, driving significant advancements in both technology and real-world applications. |
| Approach: | They present a large-scale dataset encompassing diverse open-source LLM structures and their performance across multiple benchmarks. |
| Outcome: | The proposed model validates the relationship between structural configurations and performance across multiple benchmarks and further corroborates the findings using mechanistic interpretability techniques. |
CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Understanding language acquisition in language models remains an open question, yet many benchmarks focus on grammatical acceptability, with far less attention to interpreting meanings conveyed by grammatological forms. |
| Approach: | They propose a benchmark to evaluate constructional understanding in language models using a controlled minimal-pair. |
| Outcome: | The proposed benchmarks show that understanding of constructions develops more slowly and remains limited even in large language models (LLMs). |
Probing LLMs for Joint Encoding of Linguistic Categories (2023.findings-emnlp)
Copied to clipboard
Giulio Starace, Konstantinos Papakostas, Rochelle Choenni, Apostolos Panagiotopoulos, Matteo Rosati, Alina Leidinger, Ekaterina Shutova
| Challenge: | Existing research suggests that a linguistic hierarchy emerges in large language models . little is known about how encodings of different linguistic phenomena interact within the models - and to what extent processing of linguistically-related categories relies on the same, shared model representations. |
| Approach: | They propose a framework for testing the joint encoding of linguistic categories in large language models. |
| Outcome: | The proposed framework shows that the same patterns hold across languages in multilingual LLMs. |
Creating ConLangs to Probe the Metalinguistic Grammatical Knowledge of LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | 'ConLang' is a term used to describe any artificially created language intended to be as expressive as naturally evolved human languages. |
| Approach: | They propose to use large language models to create a modular system that uses LLMs as a tool in the development of Constructed Languages. |
| Outcome: | The proposed system creates phonology, morphology and syntax, lexicon, orthography, and grammatical handbook using module-specific sets of prompts. |
LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models excel in various natural language tasks but struggle with long-horizon planning problems requiring structured reasoning. |
| Approach: | They propose to integrate large language models into AP and NLP planning frameworks by reviewing current research and identifying critical challenges and future directions. |
| Outcome: | The proposed frameworks are used to support reliable off-the-shelf AP planners. |
The better your Syntax, the better your Semantics? Probing Pretrained Language Models for the English Comparative Correlative (2022.emnlp-main)
Copied to clipboard
| Challenge: | Construction Grammar posits constructions as the central building blocks of language . human-like performance of pretrained language models on many NLP tasks has been alleged . |
| Approach: | They propose to use construction grammar to posit constructions as the central building blocks of language . they conduct experiments with three pretrained language models to examine their ability to classify and understand English comparative correlative . |
| Outcome: | The proposed models are able to recognise the English comparative correlative (CC) but fail to use its meaning. |
Language Models Use Monotonicity to Assess NPI Licensing (2021.findings-acl)
Copied to clipboard
| Challenge: | Neural language models (LMs) have become powerful approximators of human language . fewer studies have been done on what kind of formal semantic features are encoded by LMs . |
| Approach: | They propose a series of experiments that investigate the semantic knowledge of language models . they use diagnostic classifiers, linguistic acceptability tasks and a ranking method to investigate the models' inner workings. |
| Outcome: | The proposed method can be applied to LMs trained on filtered corpora and gain stronger insights into their generalizations. |
Navigating the Modern Evaluation Landscape: Considerations in Benchmarks and Frameworks for Large Language Models (LLMs) (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | General-purpose Language Models have changed the world of Natural Language Processing, if not the world itself. |
| Approach: | This tutorial will lay the foundations and explain the basics of evaluation and compare traditional methods to newly developed methods. |
| Outcome: | The tutorial assumes little familiarity with metrics, datasets, prompts and benchmarks . it will compare traditional methods to newly developed methods . |