Not all quantifiers are equal: Probing Transformer-based language models’ understanding of generalised quantifiers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent popularity of generalised quantifiers and role in linguistics and logic raises the question of how they affect transformer-based language models (TLMs) |
| Approach: | They propose to use textual entailment to assess the ability of TLMs to learn the meanings of generalised quantifiers by using a textual model-checking problem defined in a purely logical sense. |
| Outcome: | The proposed method allows the automatic construction of datasets with respect to which we can assess the ability of TLMs to learn the meanings of generalised quantifiers. |
Similar Papers
Identifying the limits of transformers when performing model-checking with natural language (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have focused on transformer models’ ability to perform reasoning on text, but the above question has not been adequately answered. |
| Approach: | They investigated the problem of model-checking with natural language to determine whether transformers can comprehend logical semantics in natural language. |
| Outcome: | The proposed model-checking problem is suited to address this issue but is untouched in natural language inference research. |
Rarely a problem? Language models exhibit inverse scaling in their predictions following few-type quantifiers (2023.findings-acl)
Copied to clipboard
| Challenge: | Current work suggests that language models deal poorly with quantifiers-they struggle to predict which quantifier is used in a given context and also perform poorly at generating appropriate continuations following logical quantifier. |
| Approach: | They propose to use 960 English sentence stimuli to build 22 autoregressive transformer models of different sizes to test their performance on ‘few’-type quantifiers. |
| Outcome: | The proposed models perform poorly on ‘few’-type quantifiers, and the larger the model, the worse its performance. |
Quantifying Generalizations: Exploring the Divide Between Human and LLMs’ Sensitivity to Quantification (2024.acl-long)
Copied to clipboard
| Challenge: | Generics are expressions used to communicate abstractions about categories . they allow for exceptions, and they are a powerful way to express knowledge about the world . |
| Approach: | They examine how large language models interpret generics to understand their meanings . they find that the presence of a generic sentence as context influences quantifiers based on the generalization . |
| Outcome: | The proposed models do not exhibit a strong sensitivity to quantification, the study finds . the results suggest that the presence of a generic sentence as context influences quantifiers . |
Unravelling the Logic: Investigating the Generalisation of Transformers in Numerical Satisfiability Problems (2025.acl-long)
Copied to clipboard
| Challenge: | Transformer models exhibit minimal scale and noise invariance, along with limited vocabulary and number invariancy. |
| Approach: | They probe the generalisation prowess of Transformer models with respect to the hitherto unexplored domain of numerical satisfiability problems. |
| Outcome: | The proposed models exhibit minimal scale and noise invariance, along with limited vocabulary and number invariancy. |
Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks (2022.naacl-main)
Copied to clipboard
| Challenge: | Quantifiers are pervasive in NLU benchmarks and their occurrence at test time is associated with performance drops. |
| Approach: | They propose a generalized quantifier NLI task to quantify their contribution to the errors of NLU models. |
| Outcome: | The proposed model is based on a generalized quantifier theory and is compared with pre-trained models. |
Some of Them Can be Guessed! Exploring the Effect of Linguistic Context in Predicting Quantifiers (P18-2)
Copied to clipboard
| Challenge: | cloze deletion test is a test that requires the learner to understand the context and vocabulary in order to identify the correct word. |
| Approach: | They collect data from human participants and test various models in a local and a global context condition to examine the role of linguistic context in predicting quantifiers. |
| Outcome: | The proposed models outperform humans in a local and global context and are only slightly better in the latter. |
On the Way to Lossless Compression of Language Transformers: Exploring Cross-Domain Properties of Quantization (2024.lrec-main)
Copied to clipboard
Nikita Martynov, Aleksei Goncharov, Gleb Kumichev, Evgeniy Egorov, Stanislav Vladimirovich Pavlov, Mikhail Sergeevich Durinov, Aleksandr Sergeevich Zuev, Egor Anatolievich Filimonov
| Challenge: | Modern Natural Language Processing models have a huge capacity, but this makes it difficult to employ. |
| Approach: | They propose a method to quantize at least 95% of Transformer weights without access to task-specific data so the drop in performance does not exceed 0.02%. |
| Outcome: | The proposed method quantizes 95% of Transformer weights and corresponding activations to INT8 without access to task-specific data so the drop in performance does not exceed 0.02%. |
Generics are not quantificational: A new path from language models to semantic theory (2026.findings-acl)
Copied to clipboard
| Challenge: | Generic sentences express generalizations that tolerate exceptions without explicitly communicating information about quantities. |
| Approach: | They compare generics and quantificational sentences to find out what quantifiers are . they argue that generics are not quantificationals, contrary to dominant views . |
| Outcome: | The proposed model recovers many semantic facts about quantifiers and their "quantificational counterparts". |
On the Ability and Limitations of Transformers to Recognize Formal Languages (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on LSTMs have not revealed their ability to model syntactic properties. |
| Approach: | They propose to build a Transformers model for a subclass of counter languages and find that their learning mechanism strongly correlates with their construction. |
| Outcome: | The proposed model generalizes well on counter languages and its learned mechanism correlates with it. |
Generics are puzzling. Can language models find the missing piece? (2025.coling-main)
Copied to clipboard
| Challenge: | Generic sentences express generalisations about the world without explicit quantification . human biases in stereotypes can be observed in language models, authors say . |
| Approach: | They analyze generic sentences to determine their quantification and quantify their implicit quantifications using language models. |
| Outcome: | The proposed model shows that generics are more context-sensitive than determiner quantifiers and express weak generalisations. |