Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate ChatGPT's causal reasoning abilities are based on pre-trained language models, but they rely on supervised training. |
| Approach: | They conduct the first comprehensive evaluation of ChatGPT’s causal reasoning capabilities using four state-of-the-art (STA) simulations. |
| Outcome: | The proposed model is not a good causal reasoner, but a great causal interpreter. |
Similar Papers
Exploring the Potential of ChatGPT on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies have demonstrated ChatGPT's remarkable few-shot, even zero-shot learning abilities when compared to other models. |
| Approach: | They quantitatively evaluate the performance of ChatGPT on inter-sentential relations such as temporal relations, causal relations, and discourse relations. |
| Outcome: | The proposed model performs well on temporal relations, causal relations, and discourse relations. |
Is ChatGPT a General-Purpose Natural Language Processing Task Solver? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in scale have enabled large language models to perform NLP tasks zero-shot . however, it is not known whether ChatGPT can serve as a generalist model that can perform many NLP jobs zero- shot. |
| Approach: | They empirically evaluate ChatGPT's zero-shot learning ability on 20 popular NLP datasets . they find it performs well on many tasks favoring reasoning abilities . |
| Outcome: | The proposed model can perform many NLP tasks zero-shot without adaptation on downstream data. |
CausalDialogue: Modeling Utterance-level Causality in Conversations (2023.findings-acl)
Copied to clipboard
| Challenge: | Despite widespread adoption, neural conversation models have yet to exhibit natural chat capabilities with humans . despite their widespread adoption in society, chatbots have yet not shown natural chat capability . |
| Approach: | They propose a causality-enhanced method to enhance the impact of causality at the utterance level in training neural conversation models. |
| Outcome: | The proposed method improves diversity and agility of loss functions and still needs improvement . the proposed method is based on a CausalDialogue dataset . |
A Multi-Level Benchmark for Causal Language Understanding in Social Media Discourse (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets focus on explicit causality in structured text, providing limited support for detecting implicit causal expressions. |
| Approach: | They propose a dataset of Reddit posts annotated across four causal tasks . they use a binary causal classification, explicit vs. implicit causality, cause–effect span extraction and causal gist generation to bridge causal detection and reasoning over informal discourse. |
| Outcome: | The proposed dataset analyzes 10,120 Reddit posts discussing public health related to the COVID-19 pandemic. |
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2022.tacl-1)
Copied to clipboard
Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, Diyi Yang
| Challenge: | causality has not had the same importance in natural language processing, says aaron e. smith . he says research on causality in NLP remains scattered across domains without unified definitions . |
| Approach: | They propose to consolidate research on causality in NLP across academic areas . they explore potential uses of causal inference to improve robustness, fairness, interpretability . |
| Outcome: | The proposed method is a unified overview of causal inference for the NLP community. |
CausalNLP Tutorial: An Introduction to Causality for Natural Language Processing (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | Establishing causal relationships is a fundamental goal of scientific research . lack of clear definitions, notations, benchmark datasets, and challenges remains . |
| Approach: | They introduce the fundamentals of causal discovery and causal effect estimation to the natural language processing audience and provide an overview of causal perspectives to NLP problems. |
| Outcome: | This tutorial introduces the fundamentals of causal discovery and causal effect estimation to the natural language processing audience and provides an overview of causal perspectives to NLP problems. |
ChatGPT Is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | acquiring and representing commonsense in machines has posed a long-standing challenge (Li et al., 2021; Zhang e t al, 2022; Zhou e al. 2023) . |
| Approach: | They use a commonsense-based LLM to evaluate ChatGPT's commonsensing abilities by analyzing 11 datasets and generating knowledge descriptions. |
| Outcome: | The proposed model can achieve good QA accuracies while still struggling with certain domains of datasets. |
CausalLink: An Interactive Evaluation Framework for Causal Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation frameworks for causal reasoning are unclear . we propose a framework that disentangles reasoning processes from confounding factors . |
| Approach: | They propose a framework that assesses the causal reasoning skill to identify correct interventions in conversational language models. |
| Outcome: | The proposed evaluation framework isolates causal capabilities from confounding effects of world knowledge and semantic cues. |
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023.findings-acl)
Copied to clipboard
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, Jimmy Huang
| Challenge: | Currently, the evaluation of large language models (LLMs) such as ChatGPT in academic datasets is difficult due to the difficulty of evaluating the generative outputs produced by this model against the ground truth. |
| Approach: | They evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in academic datasets. |
| Outcome: | The proposed model performs well on 140 tasks and generates 255K responses in these datasets. |
CausalityCheck: A Framework for Evaluating Causal Reasoning in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods fail to accurately reflect a model's causal reasoning capabilities. |
| Approach: | They propose a tool to automatically generate causal reasoning checklists to assess the causal reasoning abilities of 18 large language models. |
| Outcome: | The proposed tool assesses the causal reasoning abilities of 18 large language models. |