Thresh: A Unified, Customizable and Deployable Platform for Fine-Grained Text Evaluation (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Existing tools for fine-grained human evaluation lack adaptability to different domains or languages, or modify annotation settings according to user needs. |
| Approach: | They propose a unified platform for fine-grained evaluation that is customizable and deployable with a single YAML configuration file. |
| Outcome: | The proposed frameworks are based on a single YAML configuration file and can be easily extended to different domains or languages. |
Similar Papers
YEDDA: A Lightweight Collaborative Text Span Annotation Tool (P18-4)
Copied to clipboard
| Challenge: | Existing annotation tools do not consider post-annotation quality analysis due to inter-annotator disagreement. |
| Approach: | They propose a lightweight but efficient open-source tool for text span annotation that can be used for collaborative user annotation and administrator evaluation and analysis. |
| Outcome: | The proposed system reduces the annotation time by half compared with existing tools and the time can be compressed by 16.47% through intelligent recommendation. |
UniSumEval: Towards Unified, Fine-grained, Multi-dimensional Summarization Evaluation for LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks for summarization quality evaluation lack diverse input scenarios, focus on narrowly defined dimensions, and struggle with subjective and coarse-grained annotation schemes. |
| Approach: | They propose to use AI to help human annotations and identifie potentially hallucinogenic input texts. |
| Outcome: | The proposed benchmarks improve on existing benchmarks in terms of input diversity, granularity of human annotations, and evaluation dimensions. |
FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Existing tools to evaluate long text outputs are lacking in the field of NLP . human rating and error analysis remains a crucial component for any evaluation of long text generation. |
| Approach: | They propose a web-based toolkit to collect fine-grained error annotations for long texts . they use a taxonomy to identify errors and assign them to text spans . |
| Outcome: | The proposed tool can be used to evaluate the coherence of long generated summaries. |
ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Links (2026.eacl-long)
Copied to clipboard
| Challenge: | Using retrieval models and LLMs achieves a 73% approval rate for suggested links, more than doubling the acceptance of strong retrievers alone. |
| Approach: | They propose a domain-agnostic framework for bootstrapping sentence-level cross-document links from scratch and apply it to large-scale human-in-the-loop annotation of natural text pairs. |
| Outcome: | The proposed framework generates semi-synthetic datasets and uses them to benchmark and shortlist the best-performing methods and applies them in large-scale human-in-the-loop annotation of natural text pairs. |
Appraise Evaluation Framework for Machine Translation (C18-2)
Copied to clipboard
| Challenge: | Appraise is an open-source framework for crowd-based annotation tasks . it is used for shared tasks at the conference on machine translation and at IWSLT 2017 . |
| Approach: | They present an open-source framework for crowd-based annotation tasks . they describe the entire lifecycle of an Appraise evaluation campaign . |
| Outcome: | The proposed framework is used to run evaluation campaigns at the WMT Conference on Machine Translation and at IWSLT 2017 . it has been adopted by the translator team at Microsoft Translator for internal quality monitoring . |
Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark (2025.acl-long)
Copied to clipboard
| Challenge: | Existing MLLMs rely on commercial models such as GPT-4o for evaluations, but they are not universally accessible. |
| Approach: | They propose a task decomposition evaluation framework based on GPT-4o to automatically construct a specialized training dataset to break down the multifaceted evaluation process into simpler sub-tasks. |
| Outcome: | The proposed framework outperforms the current state-of-the-art GPT-4o evaluation framework with over 4.6% improvement in Spearman and Kendall correlations with human judgments. |
DoTAT: A Domain-oriented Text Annotation Tool (2022.acl-demo)
Copied to clipboard
| Challenge: | DoTAT is a domain-oriented text annotation tool that can reduce the time for event annotation by 19.7% . the tool supports multi-person collaborative process with automatically merging and review . |
| Approach: | They propose a domain-oriented text annotation tool called DoTAT . it provides multi-person collaborative process with automatic merging and review . |
| Outcome: | The proposed tool can reduce the time for event annotation by 19.7% compared with existing tools. |
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022.emnlp-main)
Copied to clipboard
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, Tao Yu
| Challenge: | Structured knowledge grounding (SKG) uses structured knowledge to complete user requests . since inputs and outputs of SKG tasks are heterogeneous, they have been studied separately . |
| Approach: | They propose a framework that unifies 21 SKG tasks into a text-to-text format . they use unifiedSKG to benchmark T5 with different sizes . |
| Outcome: | The proposed framework unifies 21 SKG tasks into a text-to-text format . it achieves state-of-the-art performance on almost all of the 21 tasks, the authors show . |
Evalverse: Unified and Accessible Library for Large Language Model Evaluation (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Evalverse is a library that unifies disparate evaluation tools into a single, user-friendly framework. |
| Approach: | They propose to integrate existing evaluation frameworks into a single, user-friendly framework that enables individuals with limited knowledge of artificial intelligence to request LLM evaluations and receive detailed reports. |
| Outcome: | The proposed framework can be used by individuals with limited knowledge of artificial intelligence to request and receive LLM evaluations and receive detailed reports. |
CoreCodeBench: Decoupling Code Intelligence via Fine-Grained Repository-Level Tasks (2026.acl-long)
Copied to clipboard
Lingyue Fu, Hao Guan, Bolun Zhang, Haowei Yuan, Yaoming Zhu, Lin Qiu, ZongYu Wang, Xuezhi Cao, Xunliang Cai, Weiwen Liu, Weinan Zhang, Yong Yu
| Challenge: | Existing large language models for software engineering rely on coarse-grained pass rates obscuring specific cognitive bottlenecks. |
| Approach: | They propose a repository-level benchmark that dissects coding capabilities through atomized tasks. |
| Outcome: | The proposed framework achieves a 78.55% validity yield, surpassing the 31.7% retention rate of SWE-bench-Verified. |