| Challenge: | Existing metrics for rationale evaluation focus on the association between the rationale and a label, whereas REV is more sensitive to new information in free-text rationales. |
| Approach: | They propose a metric called REV to quantify the amount of new, label-relevant information in a rationale beyond the information already available in the input or the label. |
| Outcome: | The proposed metric is consistent with human judgments on rationale evaluations and provides more sensitive measurements of new information in free-text rationales. |
Similar Papers
Measuring Association Between Labels and Free-Text Rationales (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for extractive rationales do not work as well on reasoning tasks requiring free-text rationale. |
| Approach: | They propose to use pipelines to extract rationales from input words and to use them to explain reasoning tasks. |
| Outcome: | The proposed models exhibit desirable properties for explaining commonsense question-answering and natural language inference, indicating their potential for producing faithful free-text rationales. |
RORA: Robust Free-Text Rationale Evaluation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing metrics rely on degree to which rationale supports a label, but they fail to evaluate rationales that inadvertently leak the label. |
| Approach: | They propose a RObust free-text RAtionale evaluation against label leakage that quantifies the new information supplied by a rationale to justify the label. |
| Outcome: | The proposed evaluation outperforms existing methods in evaluating human-written, synthetic, or model-generated rationales, particularly demonstrating robustness against label leakage. |
Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-text Rationales (2023.acl-long)
Copied to clipboard
Brihi Joshi, Ziyi Liu, Sahana Ramnath, Aaron Chan, Zhewei Tong, Shaoliang Nie, Qifan Wang, Yejin Choi, Xiang Ren
| Challenge: | Existing metrics like task performance of the LM generating the rationales or similarity between generated and gold rationale are not good indicators of their human utility. |
| Approach: | They propose to use a large language model to generate rationales with better human utility by estimating its conciseness and novelty. |
| Outcome: | The proposed model can measure human utility to a better extent by estimating its usefulness in answering similar unseen instances. |
Persuasiveness of Generated Free-Text Rationales in Subjective Decisions: A Case Study on Pairwise Argument Ranking (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on generating free-text rationales has focused on tasks where there is an expected factual ground truth. |
| Approach: | They analyze generated free-text rationales in tasks with subjective answers . they find open-source LLMs generate highly persuasive rationale models . |
| Outcome: | The proposed model outperforms closed-source models in pairwise argument ranking, a highly subjective task with potential for debate assistance. |
AXCEL: Automated eXplainable Consistency Evaluation using LLMs (2024.findings-emnlp)
Copied to clipboard
P Aditya Sreekar, Sahil Verma, Suransh Chopra, Abhishek Persad, Sarik Ghazarian, Narayanan Sadagopan
| Challenge: | Large Language Models (LLMs) are widely used for various tasks but evaluating the consistency of generated text remains a challenge. |
| Approach: | They propose a prompt-based consistency metric which provides explanations for consistency scores by providing detailed reasoning and pinpointing inconsistent text spans. |
| Outcome: | The proposed metric outperforms state-of-the-art metrics in summarization, free text generation and data-to-text conversion tasks by 8.7% and 6.2%. |
On Sample Based Explanation Methods for NLP: Faithfulness, Efficiency and Semantic Evaluation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for explaining "black-box" models such as Influence Functions are becoming more popular. |
| Approach: | They propose a semantic-based evaluation metric that can better align with humans’ judgment of explanations than the widely adopted diagnostic or re-training measures. |
| Outcome: | The proposed method can better align with humans’ judgment of explanations than diagnostic or re-training measures. |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
A Causal Lens for Evaluating Faithfulness Metrics (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) offer natural language explanations as an alternative to feature attribution methods for model interpretability, but they may not reflect the model’s truereasoning faithfully. |
| Approach: | They propose a testbed framework for evaluating faithfulness metrics for natural language explanations using diagnosticity and model-editing methods. |
| Outcome: | The proposed framework evaluates faithfulness metrics for natural language explanations on four tasks including fact-checking, analogy, object counting, and multi-hop reasoning. |
Identifying Reliable Evaluation Metrics for Scientific Text Revision (2025.acl-long)
Copied to clipboard
| Challenge: | Effective revision is a critical step in scientific writing, ensuring clarity, coherence, and adherence to academic standards. |
| Approach: | They propose to use ROUGE and BERTScore to assess revision quality . they also examine LLM-as-a-judge approaches to assess instruction-following revisions . |
| Outcome: | The proposed method improves the accuracy of revision tasks with and without a gold reference. |
On the Limitations of Reference-Free Evaluations of Generated Text (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study has shown that evaluation metrics which accurately estimate the quality of generated text are limited in their ability to evaluate generated text. |
| Approach: | They argue that reference-free metrics are limited in their ability to evaluate generated text . they recommend that they be used as diagnostic tools for analyzing and understanding model behavior . |
| Outcome: | The proposed evaluation metrics are limited in their ability to evaluate generated text . they can be optimized at test time, can be biased against models with similar outputs . |