Papers with NLMs
ER-Test: Evaluating Explanation Regularization Methods for Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Explanation regularization (ER) aims to improve NLM generalization by pushing the NLM’s machine rationales to align with human rationale. |
| Approach: | They propose a framework for evaluating ER models’ OOD generalization along three dimensions: unseen datasets, contrast set tests, and functional tests. |
| Outcome: | The proposed framework evaluates ER models’ OOD generalization across unseen datasets, contrast set tests, and functional tests. |
WinoLogic: A Zero-Shot Logic-based Diagnostic Dataset for Winograd Schema Challenge (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent success of neural language models on the Winograd Schema Challenge has called for further investigation of commonsense reasoning ability of these models. |
| Approach: | They propose a logic-based framework that focuses on high-quality commonsense knowledge. |
| Outcome: | The proposed framework focuses on high-quality commonsense knowledge. |
Too Big to Fail: Larger Language Models are Disproportionately Resilient to Induction of Dementia-Related Linguistic Anomalies (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that the attention mechanism in transformer-based NLMs may present an analogue to the notions of cognitive and brain reserve. |
| Approach: | They propose a bidirectional ablation method that masks attention heads to display degradation of similar magnitude to masking in smaller models. |
| Outcome: | The proposed method exhibits properties attributed to the concepts of cognitive and brain reserve in human brain studies. |
Efficient Nearest Neighbor Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Non-parametric neural language models (NLMs) learn text distributions by memorizing training data points. |
| Approach: | They propose to use an external datastore to learn from a non-parametric language model. |
| Outcome: | The proposed methods achieve up to a 6x speed-up in inference speed while retaining comparable performance. |
The Grammar-Learning Trajectories of Neural Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | In this paper, we show that neural language models with different initialization, architecture, and training data acquire linguistic phenomena in a similar order, despite their different end performance. |
| Approach: | They propose to use mutual inductive bias to study linguistic representations implicit in NLMs. |
| Outcome: | The proposed approach shows that NLMs with different initialization, architecture, and training data acquire linguistic phenomena in a similar order, despite their different end performance. |
Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation (P19-1)
Copied to clipboard
| Challenge: | Contextual embeddings address the problem of meaning conflation hampering word embeddables. |
| Approach: | They propose a method that creates sense-level embeddings with full-coverage of WordNet without recourse to explicit sense distributions or task-specific modelling. |
| Outcome: | The proposed method surpasses previous systems using powerful models and is robust when ignoring part-of-speech and lemma features. |
Coherent or Not? Stressing a Neural Language Model for Discourse Coherence in Multiple Languages (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing work on coherence assessment using NLMs focuses on properties acquired from stand-alone sentences, but their ability to model discourse and pragmatic phenomena is still unclear. |
| Approach: | They propose to use a Neural Language Model to assess coherence in multiple languages to compare models' performance and to examine their performance in a cross-language scenario. |
| Outcome: | The proposed model can model coherent and incoherent text in multiple languages and in-domain settings. |
Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for creating metaembeddings from static word embeddings have been proposed, but they are not tied to a particular downstream task. |
| Approach: | They propose a sentence-level meta-embedding learning method that takes contextualised word embedding models and learns a phrase embeddable that preserves complementary strengths of the input source NLMs. |
| Outcome: | The proposed method outperforms existing methods on semantic textual similarity benchmarks on a supervised baseline and on token-level embeddings. |
Learning Trajectories of Figurative Language for Pre-Trained Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Figures of speech and figures of language are used in everyday communication . however, this imaginative use of words requires a solid understanding of semantics and real-world knowledge. |
| Approach: | They exploit probing tasks to analyse how NLMs recognise figurative language . they find out which layers have a better comprehension of figurativ language based on pre-training data. |
| Outcome: | The proposed model can recognise hyperboles, metaphors, oxymorons and pleonasms . data show which layers have a better comprehension of figurative language . |
From Human Reading to NLM Understanding: Evaluating the Role of Eye-Tracking Data in Encoder-Based Models (2025.acl-long)
Copied to clipboard
| Challenge: | integrating eye-tracking features into Neural Language Models does not degrade downstream task performance, enhances alignment between model attention and human attention patterns, and compresses the embedding space. |
| Approach: | They used eye-gaze data from the Ghent Eye-Tracking Corpus to investigate how integrating knowledge of human reading behavior impacts Neural Language Models. |
| Outcome: | The proposed approach does not degrade downstream task performance, enhances alignment between model attention and human attention patterns, and compresses the embedding space. |