ODASim: Ordered, Distinctive and Absolute Semantic Similarity for Code Explanation Evaluation (2026.findings-acl)
Copied to clipboard
Prince Kumar, Vitobha Munigala, Jaydeep Sen, Ashish Mittal, Vishwajeet Kumar, Srikanth G. Tamilselvam
| Challenge: | Existing methods for code explanations fail to distinguish correct from partially or fully incorrect explanations and their similarity scores are poorly calibrated. |
| Approach: | They propose a model-agnostic graded fine-tuning framework that learns calibrated similarity representations between code and explanations to support fine-grained supervision and evaluation. |
| Outcome: | The proposed framework improves F1 score and ECE scores on two embedding models and reduces expected calibration error. |
Similar Papers
o-MEGA: Optimized Methods for Explanation Generation and Analysis (2025.emnlp-demos)
Copied to clipboard
| Challenge: | a growing number of transformer-based language models have created challenges for model transparency and trustworthiness. |
| Approach: | They propose a tool to automatically identify the most effective explainable AI methods . they evaluate o-mega on a post-claim matching pipeline using a curated dataset . |
| Outcome: | The proposed tool shows that the most effective explainable AI methods can be implemented in semantic matching tasks. |
CodeExp: Explanatory Code Document Generation (2022.findings-emnlp)
Copied to clipboard
Haotian Cui, Chenglong Wang, Junjie Huang, Jeevana Priya Inala, Todd Mytkowicz, Bo Wang, Jianfeng Gao, Nan Duan
| Challenge: | Existing code-to-text generation models produce only high-level code summaries that do not capture implementation-level choices essential for these scenarios. |
| Approach: | They propose a code explanation generation task that uses code docstrings to refine models. |
| Outcome: | The proposed model can generate well-structured long docstrings comparable to human-written ones. |
A Survey of the State of Explainable AI for Natural Language Processing (2020.aacl-main)
Copied to clipboard
| Challenge: | Recent years have seen significant advances in the quality of state-of-the-art models, but they have come at the expense of models becoming less interpretable. |
| Approach: | This survey examines the current state of Explainable AI within the domain of NLP . they detail the operations and explainability techniques currently available for generating explanations for NLP models . |
| Outcome: | This survey examines the state of explainable AI (XAI) within the domain of natural language processing . it focuses on the operations and explainability techniques currently available for NLP models . |
MATCH: Task-Driven Code Evaluation through Contrastive Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | GitHub Copilot generates 46% of the code on GitHub. |
| Approach: | They propose a reference-free metric that uses Contrastive Learning to generate meaningful embeddings for code and natural language task descriptions. |
| Outcome: | This paper compares the performance of a new similarity score with existing metrics. |
XMD: An End-to-End Framework for Interactive Explanation-Based Debugging of NLP Models (2023.acl-demo)
Copied to clipboard
Dong-Ho Lee, Akshen Kadakia, Brihi Joshi, Aaron Chan, Ziyi Liu, Kiran Narahari, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, Xiang Ren
| Challenge: | Existing models are susceptible to learning spurious biases that do not reflect the underlying task. |
| Approach: | They propose an open-source framework for explanation-based model debugging that allows users to provide various forms of feedback on model explanations. |
| Outcome: | The proposed framework improves model’s OOD performance on text classification tasks by up to 18%. |
XATU: A Fine-grained Instruction-based Benchmark for Explainable Text Updates (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing text editing benchmark datasets contain coarse-grained instructions and lack explainability, resulting in outputs that deviate from intended changes. |
| Approach: | They propose a benchmark specifically designed for fine-grained instruction-based explainable text editing. |
| Outcome: | The proposed benchmark incorporates fine-grained instructions and gold-standard edit explanations. |
Interpretable Text Embeddings and Text Similarity Explanation: A Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text embeddings are a fundamental component in many NLP tasks, but their interpretation and explanation remain challenging. |
| Approach: | They propose a framework for interpretable text embeddings and text similarity explanation . they characterize the main ideas, approaches, and trade-offs and discuss lessons learned . |
| Outcome: | The proposed methods are compared with existing models and compare them with existing ones. |
CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow (2024.findings-acl)
Copied to clipboard
| Challenge: | Comprising 3,402 crafted examples, our dataset is designed for both model finetuning and standalone evaluation. |
| Approach: | They propose a dataset that provides examples that include a clarified intent, code snippets associated, and an average of three related unit tests. |
| Outcome: | The proposed dataset includes 3,402 hand-written examples and 3,121 unrefined examples. |
Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language Explanations (2023.acl-long)
Copied to clipboard
| Challenge: | Human-annotated labels and explanations are critical for training explainable NLP models. |
| Approach: | They propose a metric that measures the usefulness of an explanation for model performance at both fine-tuning and inference. |
| Outcome: | The proposed metric can evaluate the quality of human-annotated explanations, while Simulatability falls short. |
SHAP-Based Explanation Methods: A Review for NLP Interpretability (2022.coling-1)
Copied to clipboard
| Challenge: | Existing models with opacity problems have been proposed to address this problem. |
| Approach: | They propose a unified local-interpretability framework with a rigorous theoretical foundation on the game-theoretic concept of Shapley values. |
| Outcome: | The proposed framework is based on the Shapley-value-based model explanations. |