Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to generate instructions using large language models require spatiotemporal awareness of multiple objects and their surroundings. |
| Approach: | They propose a tree-based evaluation metric for LLM-generated step-by-step assembly instructions that more accurately reflects spatiotemporal aspects of construction than traditional metrics such as BLEU and BERT similarity scores. |
| Outcome: | The proposed metric better correlates with manually-annotated error counts, and is more robust against artificially-constructed counterfactual examples that are specifically constructed to confound metrics that rely on textual similarity. |
Similar Papers
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for judging metrics are sensitive to the translations used for evaluation, leading to falsely confident conclusions about a metric’s efficacy. |
| Approach: | They propose a method for thresholding performance improvement under an automatic metric against human judgements by using a pairwise system ranking method. |
| Outcome: | The proposed method allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. |
How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on instruction-tuned Large Language Models has used automatic methods based on text overlap and LLM judgments as cost-effective alternatives to human evaluation. |
| Approach: | They perform a meta-evaluation of automatic methods and assess their reliability across a broad range of tasks. |
| Outcome: | The proposed method is unreliable in free-form generation tasks and cross-lingual scenarios. |
LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation metrics for natural language generation are expensive and time-consuming. |
| Approach: | They propose a framework that utilizes LLMs to generate synthetic evaluation datasets . they propose meta-correlation to measure alignment between metric rankings and human benchmarks based on synthetic data . |
| Outcome: | The proposed framework achieves meta-correlations exceeding 0.9 in multilingual QA and replaces human judgment with synthetic evaluation datasets. |
Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages (2026.acl-long)
Copied to clipboard
| Challenge: | Existing metrics have been developed and validated for English and other languages . this narrow focus leaves Indian languages largely overlooked, casting doubt on universality of current evaluation practices. |
| Approach: | They propose a large-scale benchmark that compares 26 automatic metrics with human judgments across six major Indian languages. |
| Outcome: | ITEM evaluates alignment of 26 automatic metrics with human judgments across six languages . authors: outliers exert significant impact on metric-human agreement, improve fidelity . they say the results offer critical guidance for advancing metric design and evaluation in Indian languages - a global market for machine translation and text summarization systems. |
Metric Calculating Benchmark: Code-Verifiable Complicate Instruction Following Benchmark for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent frontier-level LLMs have saturated many previously difficult benchmarks, leaving little room for further differentiation. |
| Approach: | They propose a benchmark to evaluate whether LLMs can execute string-matching NLP metrics by strictly following step-by-step instructions. |
| Outcome: | The proposed benchmarks show that they can perform step-by-step execution, instruction adherence, numerical computation, and long-range consistency in handling intermediate results. |
SemBleu: A Robust Metric for AMR Parsing Evaluation (P19-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a semantic formalism where the meaning of a sentence is encoded as a rooted, directed graph. |
| Approach: | They propose a metric that extends SMATCH to parse AMRs and does not suffer from search errors. |
| Outcome: | The proposed metric does not suffer from search errors and considers non-local correspondences in addition to local ones. |
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021.emnlp-main)
Copied to clipboard
| Challenge: | Evaluation metrics are a key ingredient for progress of text generation systems . a class of novel evaluation metrics based on BERT and its variants has been explored . |
| Approach: | They propose to disentangle BERT-based evaluation metrics along linguistic factors . they show they are sensitive to lexical overlap, just like BLEU and ROUGE . |
| Outcome: | The proposed metrics capture all aspects but are sensitive to lexical overlap, just like BLEU and ROUGE, the authors show . |
CAMIEval: Enhancing NLG Evaluation through Multidimensional Comparative Instruction-Following Analysis (2025.naacl-long)
Copied to clipboard
| Challenge: | Evaluating the quality of texts generated by language models has always been a challenging task in natural language processing (NLP). |
| Approach: | They propose a multidimensional comparative evaluation method based on instruction-following that combines relevance, factuality, and adherence with a concrete Chain-of-Thoughts process to enhance the accuracy of evaluations. |
| Outcome: | The proposed method outperforms existing methods in correlation with human evaluations on two NLG evaluation benchmarks. |
Reproducibility Issues for BERT-based Evaluation Metrics (2022.emnlp-main)
Copied to clipboard
| Challenge: | Reproducibility is of utmost concern in machine learning and natural language processing . lexical-overlap metrics are still the dominant metric in natural language generation . |
| Approach: | They ask whether results and claims from four recent BERT-based evaluation metrics can be reproduced. |
| Outcome: | The proposed metrics outperform the dominant metric, BLEU, and show that they can be reproduced. |
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models (2025.naacl-industry)
Copied to clipboard
| Challenge: | Typical evaluations of Large Language Models (LLMs) report a single accuracy metric per dataset, often derived from an optimized setup. |
| Approach: | They propose a framework for non-adversarial evaluation of large language models that evaluates models by repeatedly testing them on the same benchmarks in various setups. |
| Outcome: | The proposed framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. |