Papers with ITEM
Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages (2026.acl-long)
Copied to clipboard
| Challenge: | Existing metrics have been developed and validated for English and other languages . this narrow focus leaves Indian languages largely overlooked, casting doubt on universality of current evaluation practices. |
| Approach: | They propose a large-scale benchmark that compares 26 automatic metrics with human judgments across six major Indian languages. |
| Outcome: | ITEM evaluates alignment of 26 automatic metrics with human judgments across six languages . authors: outliers exert significant impact on metric-human agreement, improve fidelity . they say the results offer critical guidance for advancing metric design and evaluation in Indian languages - a global market for machine translation and text summarization systems. |
An Iterative Utility Judgment Framework Inspired by Philosophical Relevance via LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Relevance emphasizes the aboutness of a result to a query, while utility refers to the result’s usefulness or value to an information seeker. |
| Approach: | They propose an Iterative utiliTy judgmEnt fraMework to promote each step in Retrieval-Augmented Generation (RAG) they propose to use relevance ranking, utility judgments, and answer generation to prioritize high-utility results over low-utilitity results. |
| Outcome: | The proposed framework improves relevance, ranking, and answer generation on retrieval (TREC DL, WebAP), utility judgment task (GTI-NQ), and factoid question-answering (NQ) datasets. |