When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing theories on how humans track discourse entities are based on the idea that humans maintain explicit memory representations for each entity that encode all properties of an entity and its relation to other entities. |
| Approach: | They adapt the psycholinguistic assessment of language models paradigm to higher-level linguistic phenomena and introduce an English evaluation suite that targets the knowledge of the interactions between sentential operators and indefinite NPs. |
| Outcome: | The evaluation suite targets the knowledge of the interactions between sentential operators and indefinite NPs and the models are challenged by multiple NP's and their behavior is not systematic. |
Similar Papers
Entity Tracking in Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies on the ability of large language models to track discourse entities have not been conducted. |
| Approach: | They propose to investigate whether large language models can track entities . they first investigate whether Flan-T5, GPT-3 and GPT-3.5 can track the state of entities based on an English description of the initial state and a series of state-changing operations. |
| Outcome: | The proposed task investigates whether language models can track entities based on language descriptions and state-changing operations. |
New or Old? Exploring How Pre-Trained Language Models Represent Discourse Entities (2022.coling-1)
Copied to clipboard
| Challenge: | Recent research shows pre-trained language models learn to encode syntactic knowledge to a certain degree. |
| Approach: | They propose to investigate the information-status of entities as discourse-new or discourse-old . they use binary classification and sequence labeling to investigate their ability to encode syntactic knowledge . |
| Outcome: | The proposed models encode information on whether an entity has been introduced before or not in the discourse. |
Developmental Negation Processing in Transformer Language Models (2022.acl-short)
Copied to clipboard
| Challenge: | Negation is an important construct in language for reasoning over the truth of propositions, garnering interest from philosophy (Horn, 1989) and psycholinguistics (Zwaan, 2012). |
| Approach: | They propose to frame a natural language inference task as a problem and examine how well transformers can process negation categories. |
| Outcome: | The proposed models perform better on certain categories, suggesting clear differences in how they are processed. |
LIEDER: Linguistically-Informed Evaluation for Discourse Entity Recognition (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that large language models have basic, if imperfect, DE recognition abilities, but they lack the basic knowledge of the fundamental semantic properties that govern the introduction and subsequent reference to DEs. |
| Approach: | They propose a dataset that examines large language models’ knowledge of existence, uniqueness, plurality, and novelty. |
| Outcome: | The proposed dataset shows that large language models exhibit sensitivity to all of these properties except novelty, which demonstrates that they have yet to reach human-level language understanding abilities. |
Effective Use of Transformer Networks for Entity Tracking (D19-1)
Copied to clipboard
| Challenge: | Existing pre-trained language models for entity-related tasks are not able to handle the nuances of procedural text. |
| Approach: | They propose to use pre-trained transformer networks to track entities in procedural text by restructuring input to focus on a particular entity. |
| Outcome: | The proposed models outperform baseline models on ingredient detection in recipes and QA over scientific processes on two different tasks. |
End-to-End Neural Discourse Deixis Resolution in Dialogue (2022.emnlp-main)
Copied to clipboard
| Challenge: | Lexical overlap is a strong indicator of entity coreference, both among names and in the resolution of nominals. |
| Approach: | They propose to extend their span-based entity coreference model to exploit task-specific characteristics of discourse deixis resolution in dialogue. |
| Outcome: | The proposed model achieves state-of-the-art results on the four datasets in the CODI-CRAC 2021 shared task. |
Transformers in the loop: Polarity in neural models of language (2022.acl-long)
Copied to clipboard
| Challenge: | Recent Transformer-based language representation models (LRMs) show impressive results on practical text analysis tasks, but do they have access to complex linguistic notions? |
| Approach: | They propose to use polarity as a case study to compare metrics derived from language models to human judgments obtained in psycholinguistic experiments. |
| Outcome: | The proposed model is more accurate than linguistic theory predictions for polarity, and allows us to use language models to discover new insights into natural language grammar beyond existing linguistic theories. |
Context Matters: A Pragmatic Study of PLMs’ Negation Understanding (2022.acl-long)
Copied to clipboard
| Challenge: | In linguistics, there are two main perspectives on negation: a semantic and a pragmatic view. |
| Approach: | They propose to use transformer-based pre-trained language models to study negation understanding using a pragmatic paradigm. |
| Outcome: | The proposed transformer-based model outperforms the human benchmark at NLU and GLUE, and the results are much more optimistic than previous studies. |
What Context Features Can Transformer Language Models Use? (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. |
| Approach: | They propose to use lexical and structural information to ablate usable information in transformer language models. |
| Outcome: | The proposed model improves when conditioning on contexts of thousands of previous tokens. |
Dynamic Gazetteer Integration in Multilingual Models for Cross-Lingual and Cross-Domain Named Entity Recognition (2022.naacl-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) models trained on CoNLL do not transfer well to other domains, even within the same language. |
| Approach: | They propose a token-level gating layer to augment pre-trained multilingual transformers with gazetteers containing named entities (NE) from a target language or domain. |
| Outcome: | The proposed model improves on cross-lingual transfer with an F1 score of 92.92 for English and an average of 89.43 across all languages in CoNLL. |