Papers with NLM
CharBERT: Character-aware Pre-trained Language Model (2020.coling-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) construct word representations at subword level with Byte-Pair Encoding (BPE) or its variations . but these methods split a word into subword units and make it incomplete and fragile . |
| Approach: | They propose a character-aware pre-trained language model to tackle OOV problems . they construct contextual word embedding for each token from sequential character representations . |
| Outcome: | The proposed model improves on the existing models on multiple NLP benchmarks. |
Linguistic Profiling of a Neural Language Model (2020.coling-main)
Copied to clipboard
| Challenge: | Neural Language Models (NLMs) have become a central component in NLP systems over the last few years, showing outstanding performance and improving the state-of-the-art on many tasks. |
| Approach: | They use a wide set of probing tasks to investigate how a Neural Language Model learns linguistic properties before and after a fine-tuning process. |
| Outcome: | The proposed model can encode a wide range of linguistic characteristics but loses this information when trained on specific downstream tasks. |
ER-Test: Evaluating Explanation Regularization Methods for Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Explanation regularization (ER) aims to improve NLM generalization by pushing the NLM’s machine rationales to align with human rationale. |
| Approach: | They propose a framework for evaluating ER models’ OOD generalization along three dimensions: unseen datasets, contrast set tests, and functional tests. |
| Outcome: | The proposed framework evaluates ER models’ OOD generalization across unseen datasets, contrast set tests, and functional tests. |
On the Nature of BERT: Correlating Fine-Tuning and Linguistic Competence (2022.coling-1)
Copied to clipboard
| Challenge: | Several studies on the interpretation of Neural Language Models (NLMs) focus on the linguistic generalization abilities of pre-trained models, but little attention is paid to how the linguistic knowledge of the models changes during fine-tuning. |
| Approach: | They propose to examine whether a wide range of linguistic phenomena are forgotten during fine-tuning and whether it is possible to predict the fine- tuned accuracy solely relying on the assessed linguistic competence. |
| Outcome: | The proposed model can predict the evolution of written language competence of native language learners based on the assessed linguistic competence. |
The Grammar-Learning Trajectories of Neural Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | In this paper, we show that neural language models with different initialization, architecture, and training data acquire linguistic phenomena in a similar order, despite their different end performance. |
| Approach: | They propose to use mutual inductive bias to study linguistic representations implicit in NLMs. |
| Outcome: | The proposed approach shows that NLMs with different initialization, architecture, and training data acquire linguistic phenomena in a similar order, despite their different end performance. |
Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation (P19-1)
Copied to clipboard
| Challenge: | Contextual embeddings address the problem of meaning conflation hampering word embeddables. |
| Approach: | They propose a method that creates sense-level embeddings with full-coverage of WordNet without recourse to explicit sense distributions or task-specific modelling. |
| Outcome: | The proposed method surpasses previous systems using powerful models and is robust when ignoring part-of-speech and lemma features. |
Coherent or Not? Stressing a Neural Language Model for Discourse Coherence in Multiple Languages (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing work on coherence assessment using NLMs focuses on properties acquired from stand-alone sentences, but their ability to model discourse and pragmatic phenomena is still unclear. |
| Approach: | They propose to use a Neural Language Model to assess coherence in multiple languages to compare models' performance and to examine their performance in a cross-language scenario. |
| Outcome: | The proposed model can model coherent and incoherent text in multiple languages and in-domain settings. |