Counterfactual reasoning: Testing language models’ understanding of hypothetical scenarios (2023.acl-short)
Copied to clipboard
| Challenge: | Existing language models have demonstrated remarkable improvements in downstream tasks, but it remains unclear to what extent they are consequences of correlation with linguistic heuristics versus robust reasoning about causal relations grounded on understanding of world knowledge. |
| Approach: | They propose to test language models with counterfactual conditionals to test their ability to distinguish hypothetical scenarios from reality. |
| Outcome: | The proposed model overrides real-world knowledge in counterfactual scenarios, but most models are driven by lexical cues. |
Similar Papers
Empowering Language Understanding with Counterfactual Reasoning (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for language understanding use the recognized patterns in the testing phase that are inherently different from us humans who have counterfactual thinking. |
| Approach: | They propose a counterfactual Reasoning Model which mimics counterfactive thinking by learning from few counterffact samples. |
| Outcome: | The proposed model can detect and make predictions from textual patterns . it can also detect negative sarcastic puns by comparing them with imaginations . |
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2024.naacl-long)
Copied to clipboard
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, Yoon Kim
| Challenge: | Recent language models possess impressive performance across a wide range of tasks . however, they often rely on narrow, non-transferable procedures for task-solving . |
| Approach: | They propose to evaluate language models using "counterfactual" task variants that deviate from standard tasks. |
| Outcome: | The proposed framework shows that language models perform better on a wide range of tasks compared to the default conditions. |
What if This Modified That? Syntactic Interventions with Counterfactual Embeddings (2021.findings-acl)
Copied to clipboard
| Challenge: | Prior art aims to uncover meaningful properties within model representations, but it is unclear how faithfully such probes portray information that the models actually use. |
| Approach: | They propose a technique for generating counterfactual embeddings within models . they produce evidence that some models use a tree-distancelike representation of syntax . |
| Outcome: | The proposed technique produces evidence that some models use tree-distancelike representations of syntax in downstream prediction tasks. |
Prompting Large Language Models for Counterfactual Generation: An Empirical Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have made remarkable progress in a wide range of natural language understanding and generation tasks, but their ability to generate counterfactuals has not been examined systematically. |
| Approach: | They propose a framework to evaluate LLMs' ability to generate counterfactuals based on key factors including intrinsic properties and prompt design. |
| Outcome: | The proposed framework examines the strengths and weaknesses of large language models (LLMs) and identifies factors that influence their ability to generate counterfactuals. |
CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models (2022.lrec-1)
Copied to clipboard
| Challenge: | CRASS data set and benchmark provide novel test scheme to evaluate large language models . authors present and explain the CRAS data set, a novel basis to test reasoning and natural language understanding of LLMs . |
| Approach: | They introduce a new test scheme utilizing questionized counterfactual conditionals to evaluate large language models. |
| Outcome: | The proposed model sets out to be the most powerful and valid tool to evaluate large language models. |
Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generation (2026.acl-long)
Copied to clipboard
Qianli Wang, Van Bach Nguyen, Yihong Liu, Fedor Splitt, Nils Feldhus, Christin Seifert, Hinrich Schuetze, Sebastian Möller, Vera Schmitt
| Challenge: | Large language models excel at generating English counterfactuals but their effectiveness in generating multilingual counterfacts remains unclear. |
| Approach: | They conduct automatic evaluations on both directly generated and derived counterfactuals in six languages and find that cross-lingual perturbations follow common strategic principles. |
| Outcome: | The proposed models show that translation-based counterfactuals offer higher validity than their directly generated counterparts, but still fall short of matching the quality of the original English counterf actuals. |
Do LLMs Behave as Claimed? Investigating How LLMs Follow Their Own Claims using Counterfactual Questions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation frameworks rely on curated datasets that, once public, may be accessed by newer LLMs. |
| Approach: | They propose a framework that generates counterfactual questions and answers from existing evaluation datasets and uses them to evaluate LLMs. |
| Outcome: | The proposed evaluation framework reduces the risk of data leakage by allowing the LLMs to respond to counterfactual questions and verify their claims. |
Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals (2024.findings-emnlp)
Copied to clipboard
Yanai Elazar, Bhargavi Paranjape, Hao Peng, Sarah Wiegreffe, Khyathi Chandu, Vivek Srikumar, Sameer Singh, Noah Smith
| Challenge: | Existing studies have found that datasets with paired inputs are prone to spurious correlations, resulting in models trained only on those outperform chance. |
| Approach: | They propose a counterfactual attentiveness test to measure reliance on spurious correlations by replacing part of the input with its counterpart from a different example. |
| Outcome: | The proposed method improves models' attentiveness on ten datasets spanning four tasks: natural language inference, reading comprehension, paraphrase detection, and visual & language reasoning. |
Large Language Models for Psycholinguistic Plausibility Pretesting (2024.findings-eacl)
Copied to clipboard
| Challenge: | Psycholinguists typically use language models to create controlled materials . plausibility judgments are often based on coarse-grained judgements, but fine-grounded ones do not . |
| Approach: | They investigate whether Language Models can be used to generate plausibility judgments . they find that plausible judgements from LMs are highly related to human judgements - whereas other LM models are not . |
| Outcome: | The proposed language models can generate plausibility judgments from human evaluators . the proposed models do not provide satisfactory discriminative power . |
Counterfactual-Consistency Prompting for Relative Temporal Understanding in Large Language Models (2025.acl-short)
Copied to clipboard
| Challenge: | Existing work has highlighted that large language models lack temporal reasoning abilities, especially when attempting to infer temporal relationships without relying on absolute time indicators. |
| Approach: | They propose a method that generates counterfactual questions and enforces collective constraints, enhancing the model’s consistency. |
| Outcome: | The proposed method shows significant improvements in event ordering for explicit and implicit events and temporal commonsense understanding. |