| Challenge: | Recent studies have shown that low-precision methods can improve performance, but they introduce high variability in the results based on tie resolution. |
| Approach: | They propose a retrieval evaluation protocol designed to reduce tie variation . high-precision scoring and tie-aware retrieval metrics are proposed to reduce this variability . |
| Outcome: | The proposed retrieval evaluation protocol reduces tie-induced instability and recovers expected scores and ranges on 12 retrieval datasets. |
Similar Papers
Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing variants of Kendall's tau have weaknesses in handling of ties, and in some situations can even be gamed. |
| Approach: | They propose to use Kendall's tau to meta-evaluate machine translation evaluation metrics with a version of pairwise accuracy that gives metrics credit for correctly predicting ties and a tie calibration procedure that automatically introduces ties into metric scores. |
| Outcome: | The proposed model gives metrics credit for correctly predicting ties and introduces ties into metric scores, enabling fair comparison between metrics that do and do not predict ties. |
Confidence and Stability of Global and Pairwise Scores in NLP Evaluation (2025.acl-srw)
Copied to clipboard
| Challenge: | Modern natural language processing benchmarks are often represented as pairwise comparison leaderboards, such as LMSYS Arena. |
| Approach: | They investigate the strengths and weaknesses of global scores and pairwise comparisons to aid decision-making in selecting appropriate model evaluation strategies. |
| Outcome: | The proposed method underestimates strong models with rare errors or low confidence, while relying on global scores can be more effective. |
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models (2025.naacl-industry)
Copied to clipboard
| Challenge: | Typical evaluations of Large Language Models (LLMs) report a single accuracy metric per dataset, often derived from an optimized setup. |
| Approach: | They propose a framework for non-adversarial evaluation of large language models that evaluates models by repeatedly testing them on the same benchmarks in various setups. |
| Outcome: | The proposed framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. |
Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks focus on pairwise matching, ignoring robustness . current models exhibit frustrating degradation, with a maximum drop of 23.43 F1 score . |
| Approach: | They propose a benchmark that simulates the evaluation of open information extraction models in the real world . they perform experiments on typical models published in the last decade and a representative large language model . |
| Outcome: | The proposed model is rated robust on a knowledge-invariant clique with different syntactic and expressive forms. |
Numbers Matter! Bringing Quantity-awareness to Retrieval Systems (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Quantitative information is important for understanding documents and interpreting them. |
| Approach: | They propose two quantity-aware ranking techniques that rank both quantity and textual content . they use available retrieval systems to incorporate quantity information into queries . |
| Outcome: | The proposed methods can rank both quantity and textual content, either jointly or independently. |
Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarking (2023.findings-acl)
Copied to clipboard
Keshav Santhanam, Jon Saad-Falcon, Martin Franz, Omar Khattab, Avi Sil, Radu Florian, Md Arafat Sultan, Salim Roukos, Matei Zaharia, Christopher Potts
| Challenge: | Neural information retrieval (IR) systems have progressed rapidly in recent years . many IR benchmarks focus on downstream task accuracy, concealing costs incurred . |
| Approach: | They propose to include efficiency considerations on IR benchmarks to help drive progress . eral et al. propose to incorporate query latency and cost budgets into evaluation . |
| Outcome: | a new study shows that the best IR system varies according to how efficiency considerations are chosen and weighed . the proposed benchmarks would allow for more thorough exploration of possible system designs . |
From Confidence to Collapse in LLM Factual Robustness (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation methods focus on performance-based metrics, often investigating from the perspective of prompt perturbations, which captures only the externally triggered side of knowledge robustness. |
| Approach: | They propose a method to measure factual robustness from the perspective of the generation process by analyzing token distribution entropy and temperature scaling sensitivity. |
| Outcome: | The proposed method measures factual robustness from the perspective of the generation process and entropy and temperature scaling sensitivity. |
A Tale of Evaluating Factual Consistency: Case Study on Long Document Summarization Evaluation (2025.findings-acl)
Copied to clipboard
| Challenge: | Despite the recent progress for summarization models in producing fluent summaries, they still encounter challenges when long sequences of generated texts and inputs (over thousands of words) need to be evaluated. |
| Approach: | They conduct a systematic analysis of factual-consistency evaluation systems across four long-document datasets and examine the relationship between sentence-level and summary-level model performance. |
| Outcome: | The proposed models can achieve higher recall in error detection for older summaries, yet struggle with false positives and fine-grained error detection. |
Threshold-driven Pruning with Segmented Maximum Term Weights for Approximate Cluster-based Sparse Retrieval (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using rank score thresholding, sparse retrieval skips the index at cluster and document levels. |
| Approach: | They propose a pruning control scheme with a probabilistic guarantee on rank-safeness competitiveness. |
| Outcome: | The proposed pruning control scheme improves accuracy and safeness while delivering low latency on single-threaded CPU. |
Deep Relevance Ranking Using Enhanced Document-Query Interactions (D18-1)
Copied to clipboard
| Challenge: | Document relevance ranking is the task of ranking documents from a large collection using the query and the text of each document only. |
| Approach: | They propose to use convolutional n-gram matching to inject rich context-sensitive encodings into their models, inspired by PACRR's convolution-based ngram matching features. |
| Outcome: | The proposed models outperform baselines, DRMM, and PACRR on the BIOASQ and TREC ROBUST questions and document inputs. |