Do large language models and humans have similar behaviours in causal inference with script knowledge? (2024.starsem-1)
Copied to clipboard
| Challenge: | Recent studies show pre-trained language models have superior language understanding abilities, including zero-shot causal reasoning. |
| Approach: | They used a script-based story to manipulate event B in a story which causally depends on a previous event A. |
| Outcome: | The results show that only recent LLMs, like GPT-3 or Vicuna, correlate with human behavior in the A B condition. |
Similar Papers
What do Large Language Models Learn about Scripts? (2022.starsem-1)
Copied to clipboard
| Challenge: | Script Knowledge is important for language understanding but expensive to produce manually and difficult to induce from text due to reporting bias. |
| Approach: | They propose a pipeline-based script induction framework which can generate good quality ESDs for unseen scenarios. |
| Outcome: | The proposed framework produces good quality ESDs for unseen scenarios, but manual evaluation shows there is room for improvement. |
Generating Hypothetical Events for Abductive Inference (2021.starsem-1)
Copied to clipboard
| Challenge: | Abductive reasoning is inference to the best explanation given an incomplete set of observations about everyday situations. |
| Approach: | They propose a model that generates what could happen next from a hypothetical scenario and then proposes the most plausible explanation from varying hypothetical scenarios. |
| Outcome: | The proposed model improves over previous vanilla pre-trained models fine-tuned on Abductive NLI. |
Lexical Substitution as Causal Language Modeling (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing methods for lexical substitution task lacks autoregressive decoding capabilities. |
| Approach: | They propose a framework that uses causal language modeling (CLM) for lexical substitution task. |
| Outcome: | The proposed system outperforms GeneSis, the best previously published supervised LST method. |
Syntax and Semantics Meet in the “Middle”: Probing the Syntax-Semantics Interface of LMs Through Agentivity (2023.starsem-1)
Copied to clipboard
| Challenge: | a recent study examined how large language models handle interactions in meaning across words and larger syntactic forms. |
| Approach: | They propose to use a dataset to examine the linguistic properties of optionally transitive English verbs to examine their agentivity. |
| Outcome: | The proposed model outperforms all other models in the evaluation dataset . the results are better correlated with human judgements than syntactic and semantic corpus statistics . |
Language models are not naysayers: an analysis of language models on negation benchmarks (2023.starsem-1)
Copied to clipboard
| Challenge: | Negation has been shown to be a major bottleneck for masked language models, such as BERT, but whether this finding still holds for larger-sized auto-regressive language models has not been studied comprehensively. |
| Approach: | They evaluate the ability of current-generation auto-regressive language models to handle negation using a wide range of benchmarks and models. |
| Outcome: | The proposed models are compared against a wide range of negation benchmarks and show that they are insensitive to negation, inability to capture the lexical semantics of negations, and failure to reason under negation. |
AnaLog: Testing Analytical and Deductive Logic Learnability in Language Models (2022.starsem-1)
Copied to clipboard
| Challenge: | Existing approaches to NLP tasks rely on pre-trained language models, but some do not. |
| Approach: | They propose a natural language inference task to test pre-trained language models for logical reasoning capabilities. |
| Outcome: | The proposed language model performs better than other models across logical connectives and reasoning domains, but is sensitive to lexical and syntactic variations in the realisation of logical statements. |
True Detective: A Deep Abductive Reasoning Benchmark Undoable for GPT-3 and Challenging for GPT-4 (2023.starsem-1)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated solid zero-shot reasoning capabilities, which is reflected in their performance on the current test tasks. |
| Approach: | They propose a benchmark consisting of 191 long-form mystery narratives constructed as detective puzzles. |
| Outcome: | The proposed benchmark outperforms random models on the current test tasks while state-of-the-art models only solve 38% of puzzles. |
Exploring Factual Entailment with NLI: A News Media Study (2024.starsem-1)
Copied to clipboard
| Challenge: | Recent studies have focused on the relationship between factuality and Natural Language Inference (NLI). |
| Approach: | They propose a novel annotation scheme that models factual rather than textual entailment and use it to annotate a dataset of naturally occurring sentences from news articles. |
| Outcome: | The proposed annotation scheme can be used to model factual relationships on a dataset of naturally occurring sentences from news articles. |
Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge (2021.starsem-1)
Copied to clipboard
Paolo Pedinotti, Giulia Rambelli, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, Philippe Blache
| Challenge: | Prior work has explored the ability of computational models to predict word semantic fit with a given predicate. |
| Approach: | They compare Transformers Language Models to SDM to assess their performance . they found that TLMs do not capture important aspects of event knowledge . people can discriminate between typical and atypical events, they say . |
| Outcome: | The proposed models can achieve comparable performance to SDM, but they lack important aspects of event knowledge. |
Probing neural language models for understanding of words of estimative probability (2023.starsem-1)
Copied to clipboard
| Challenge: | Words of Estimative Probability (WEP) are phrases used to express the plausibility of a statement. |
| Approach: | They propose to use a UNLI dataset to assess language models' ability to process WEPs. |
| Outcome: | The proposed model can accurately capture the consensual probability level associated with each WEP. |