LIBERTy: A Causal Framework for Benchmarking Concept-Based Explanations of LLMs with Structural Counterfactuals (2026.findings-acl)
Copied to clipboard
| Challenge: | Concept-based explanations quantify how high-level concepts influence model behavior . existing benchmarks rely on costly human-written counterfactuals that serves as imperfect proxy . |
| Approach: | They propose a framework for constructing datasets containing structural counterfactual pairs . they use a structured Causal Model to generate a concept-based explanation . |
| Outcome: | The proposed framework compares concept-based explanations to causal effects estimated from counterfactuals. |
Similar Papers
Evaluating Readability and Faithfulness of Concept-based Explanations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating concepts from different perspectives lack a unified formalization. |
| Approach: | They propose a formal definition of concepts generalizing to diverse concept-based explanations’ settings and apply it to other types of explanations or tasks. |
| Outcome: | Extensive experimental analysis was carried out to determine the evaluation measures for explanation evaluation measures. |
A Causal Lens for Evaluating Faithfulness Metrics (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) offer natural language explanations as an alternative to feature attribution methods for model interpretability, but they may not reflect the model’s truereasoning faithfully. |
| Approach: | They propose a testbed framework for evaluating faithfulness metrics for natural language explanations using diagnosticity and model-editing methods. |
| Outcome: | The proposed framework evaluates faithfulness metrics for natural language explanations on four tasks including fact-checking, analogy, object counting, and multi-hop reasoning. |
Bridging Internal Consistency and External Alignment: A Causal and Dynamic Interpretability Framework for LLM Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing interpretability methods focus on internal and external aspects of the model . existing explanations often focus on surface correlations or static dependencies . |
| Approach: | They propose a causal and dynamic interpretability framework for Large Language Models . they characterize backdoor-adjusted causal effects of generated prefix and prompt . |
| Outcome: | The proposed framework provides a unified causal view of internal consistency and external alignment in LLM generation dynamics. |
Faithful Serum: Mitigating the Faithfulness Gap in Textual Explanations of LLM Decisions via Attribution Guidance (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work has focused on generating convincing rationales that appear to be subjectively faithful, but it remains unclear whether these explanations are epistemic faithful. |
| Approach: | They propose a method that enhances epistemic faithfulness by guiding explanation generation through attention-level interventions, informed by token-level heatmaps. |
| Outcome: | The proposed method significantly improves epistemic faithfulness across multiple models, benchmarks, and prompts. |
Language Models as Causal Effect Generators (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using sequence-driven structural causal models (SD-SCMs) we characterize how SD-SCAMs enables sampling from observational, interventional, and counterfactual distributions according to the desired causal structure. |
| Approach: | They propose a sequence-driven structural causal model that uses language models to parameterize a structural causal system based on a user-specified DAG. |
| Outcome: | The proposed method outperforms state-of-the-art methods and can underpin auditing of language models for (un)desirable causal effects, such as misinformation or discrimination. |
Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence (2026.findings-acl)
Copied to clipboard
Kaijie Mo, Siddhartha Venkatayogi, Chantal Shaib, Ramez Kouzy, Wei Xu, Byron C Wallace, Junyi Jessy Li
| Challenge: | Existing models are overwhelmingly accurate when presented with counterfactual medical evidence . prior work explored conflicts between context and LLM parametric knowledge in the general domain . |
| Approach: | They construct a counterfactual medical QA dataset that requires models to answer clinical comparison questions with evidence from randomized controlled trials. |
| Outcome: | The proposed model overemphasizes the latter, and the model overestimates the latter. |
Explaining Language Model Predictions with High-Impact Concepts (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to explain large language models (LLMs) are mostly correlational and lack causal features due to compositional nature of languages. |
| Approach: | They propose a framework to provide impact-aware explanations for large language models that are robust to feature changes and influential to the model’s predictions. |
| Outcome: | The proposed explanations improve on real and synthetic tasks and are robust to feature changes and influential to the model’s predictions. |
Can LLMs Explain Themselves Counterfactually? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Explanations are an important tool for gaining insights into model behavior, calibrating user trust, and ensuring compliance. |
| Approach: | They propose to use self-explanation to prompt models to explain outputs . they find that LLMs struggle to generate SCEs - their prediction often does not agree with their own counterfactual reasoning. |
| Outcome: | The proposed methods can generate SCEs across families, sizes, temperatures, and datasets. |
ConSim: Measuring Concept-Based Explanations’ Effectiveness with Automated Simulatability (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation metrics focus only on the quality of the induced space of possible concepts, neglecting the latter. |
| Approach: | They propose to use large language models as simulators to approximate the evaluation and report various analyses to make such approximations reliable. |
| Outcome: | The proposed framework allows for scalable and consistent evaluation across models and datasets. |
MDC-Bench: A Multidisciplinary Causal Benchmark Based on Causal Structures for Evaluating Large Language Models (2026.findings-acl)
Copied to clipboard
Peng Wang, Yuxiong Yan, Xiao Ding, Kai Xiong, Bibo Cai, Chao Peng, Yutai Hou, Dandan Tu, Bing Qin, Ting Liu
| Challenge: | Existing causal datasets focus on the commonsense domain, but LLMs perform poorly when answering complex questions. |
| Approach: | They propose a multidisciplinary causal evaluation benchmark to assess LLMs' knowledge and skills. |
| Outcome: | The proposed model improves in domain specialization, structural diversity, and task complexity. |