| Challenge: | Existing evidence that deep natural language understanding models do not learn systematically is lacking. |
| Approach: | They examine whether deep natural language understanding models exhibit systematicity . they find that network architectures can generalize non-systematically . |
| Outcome: | The proposed model generalizes non-systematically, but is unsatisfactory, the authors argue . they show that the current state-of-the-art models do not generalize systematically . |
Similar Papers
Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language? (2020.acl-main)
Copied to clipboard
| Challenge: | Despite the success of language models using neural networks, it remains unclear to what extent neural models have the generalization ability to perform inferences. |
| Approach: | They propose a method to evaluate whether neural models can learn systematicity of monotonicity inference in natural language. |
| Outcome: | The proposed method shows that neural models can perform inferences on unseen combinations of lexical and logical phenomena when syntactic structures are similar between training and test sets. |
The Learnability of Model-Theoretic Interpretation Functions in Artificial Neural Networks (2026.findings-acl)
Copied to clipboard
| Challenge: | Entity vectors improve scores on basic event, while gated architectures benefit most. |
| Approach: | They extend entity-level semantic representations, modern architectures, principled competing event generation, extended systematicity tests and a two-dimensional difficulty analysis disaggregating results by modifier complexity. |
| Outcome: | The proposed model-theoretic interpretation functions generalize systematically to out-of-training-sample sentences. |
A Systematic Assessment of Syntactic Generalization in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on syntactic knowledge models has not provided a clear picture of the properties required to produce proper syntaktic generalizations. |
| Approach: | They propose to evaluate syntactic knowledge of language models by varying model architectures . they find substantial differences in syntaktic generalization performance by model architecture . |
| Outcome: | The proposed model architectures outperform other architectures on a set of 34 English-language syntactic test suites. |
Exploring Transitivity in Neural NLI Models through Veridicality (2021.eacl-main)
Copied to clipboard
| Challenge: | Despite recent success of deep neural networks in natural language processing, the extent to which they can demonstrate human-like generalization capacities remains unclear. |
| Approach: | They propose an analysis method to evaluate whether models can draw inferences composed of veridical inference and arbitrary inference types. |
| Outcome: | The proposed model performs poorly on transitivity inference tasks, suggesting it lacks generalization capacity for drawing composite inferences from training examples. |
Meaning to Form: Measuring Systematicity as Information (P19-1)
Copied to clipboard
| Challenge: | A longstanding debate in semiotics centers on the relationship between linguistic signs and their corresponding semantics: is there an arbitrary relationship between word forms and their meaning, or does some systematic phenomenon pervade? |
| Approach: | They propose to quantify the systematicity of the sign using mutual information and recurrent neural networks to examine 106 languages. |
| Outcome: | The proposed model reduces entropy in a word form conditioned on its semantic representation and recovers English examples of systematic affixes. |
Systematic Generalization in Language Models Scales with Information Entropy (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for assessing compositional behavior are unclear on how to measure the difficulty of a systematic generalization problem. |
| Approach: | They propose a framework for measuring entropy in a sequence-to-sequence task and a method for measuring it. |
| Outcome: | The proposed framework scales with the entropy of the distribution of component parts in the training data. |
Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on robustness-like metamorphic relations, which limit the scope of linguistic properties they can test. |
| Approach: | They propose three new classes of metamorphic relations which address the properties of systematicity, compositionality and transitivity. |
| Outcome: | The proposed methods show that metamorphic models do not always behave according to expected linguistic properties. |
SETI: Systematicity Evaluation of Textual Inference (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-trained language models (PLMs) have shown remarkable performance on this task, but little is known about their ability to address compositional generalization. |
| Approach: | They propose a benchmark to evaluate pre-trained language models' systematicity in the domain of textual inference. |
| Outcome: | The proposed benchmark evaluates pre-trained language models on six widely used PLMs. |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020.acl-main)
Copied to clipboard
| Challenge: | a priori, large neural language models are described as understanding or capturing meaning on tasks that are ostensibly meaningsensitive. |
| Approach: | They argue that a system trained only on form has no way to learn meaning . they argue that this is due to a misunderstanding of the relationship between form and meaning - which is a misconception in NLP . |
| Outcome: | The proposed model can't learn meaning because it only uses form as training data, the authors argue . they argue that a clear understanding of the distinction between form and meaning will guide the field towards better science around natural language understanding. |