Papers with evaluator
Designing Precise and Robust Dialogue Response Evaluators (2020.acl-main)
Copied to clipboard
| Challenge: | Existing automated dialogue response evaluators have only moderate correlation with human judgement and are not robust. |
| Approach: | They propose to build a reference-free dialogue response evaluator that exploits the power of semi-supervised training and pretrained (masked) language models. |
| Outcome: | The proposed model achieves strong correlation with human judgement and generalizes robustly to diverse responses and corpora. |
Generate, Evaluate, and Select: A Dialogue System with a Response Evaluator for Diversity-Aware Response Generation (2022.naacl-srw)
Copied to clipboard
| Challenge: | Existing dialogue systems that generate fluent responses are difficult to evaluate due to the one-to-many nature of dialogue, which means the existence of multiple appropriate responses is not appropriate. |
| Approach: | They propose a generator-evaluator model that evaluates multiple responses generated by a response generator and selects the best response by an evaluators. |
| Outcome: | The proposed model is compared with a baseline system and its outputs were judged to be better than the baseline system. |
EasyJudge: an Easy-to-use Tool for Comprehensive Response Evaluation of LLMs (2025.coling-demos)
Copied to clipboard
| Challenge: | Existing open-source evaluation models lack a user-friendly visualization tool and are not optimized for accelerated model inference. |
| Approach: | They propose to use open-source evaluation models to evaluate language model responses. |
| Outcome: | The proposed model is lightweight, precise, efficient, and user-friendly, with an intuitive visualization interface for ease of deployment and use. |
Investigating Table-to-Text Generation Capabilities of Large Language Models in Real-World Information Seeking Scenarios (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Existing table-to-text generation techniques that transform complex tabular data into comprehensible narratives are lacking in real-world applications. |
| Approach: | They investigate the table-to-text capabilities of different LLMs using four datasets within two real-world information seeking scenarios. |
| Outcome: | The proposed models can generate table-to-text data in two real-world information seeking scenarios and perform better than existing models. |
A Language Model based Evaluator for Sentence Compression (P18-2)
Copied to clipboard
| Challenge: | Existing methods to produce readable sentence compression are based on machine learning or syntactic tree-based approaches. |
| Approach: | They propose a language-model-based evaluator for deletion-based sentence compression . they propose deleting operations on source sentences to obtain best target compression based on the proposed model . |
| Outcome: | The proposed model generates more readable compression comparable to strong baselines. |
Diversity and Consistency: Exploring Visual Question-Answer Pair Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing tasks to generate question-answer pairs from visual images are under-explored. |
| Approach: | They propose a task that targets question-answer pair generation from visual images. |
| Outcome: | The proposed model can generate diverse or consistent QAPs on two benchmarks. |
Conversational Education at Scale: A Multi-LLM Agent Workflow for Procedural Learning and Pedagogic Quality Assessment (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on large language models lacks scalability and assesses pedagogic quality. |
| Approach: | They propose a multi-agent workflow leveraging large language models to simulate interactive teaching-learning conversations. |
| Outcome: | The proposed workflow integrates teacher and learner agents, an interaction manager, and an evaluator to facilitate procedural learning and assess pedagogic quality. |
Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing (2023.acl-long)
Copied to clipboard
| Challenge: | Existing work on interactive semantic parsing relies on human annotations to train a model . prior work relied on human-annotated feedback data, which is prohibitively expensive and not scalable . |
| Approach: | They propose a task of simulating NL feedback for interactive semantic parsing . they propose evaluators to assess the quality of the simulated feedback . |
| Outcome: | The proposed simulator can generate high-quality NL feedback to boost the error correction ability of a specific parser. |
GEAR: A Scalable and Interpretable Evaluation Framework for RAG-Based Car Assistant Systems (2025.emnlp-industry)
Copied to clipboard
Niloufar Beyranvand, Hamidreza Dastmalchi, Aijun An, Heidar Davoudi, Winston Chan, Ron DiCarlantonio
| Challenge: | Large language models (LLMs) increasingly power car assistants, but evaluating response quality remains a challenge. |
| Approach: | They propose a framework that uses large language models as evaluators to compare assistant responses against ground-truth counterparts. |
| Outcome: | The proposed framework compares assistant responses against ground-truth counterparts, assessing coverage, correctness, and other dimensions of answer quality. |
Automating Legal Interpretation with LLMs: Retrieval, Generation, and Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | a novel framework for automated legal interpretation is proposed to alleviate the burden on legal experts. |
| Approach: | They propose a framework for automated legal interpretation that uses large language models to extract concept-related information and interpret legal concepts. |
| Outcome: | The proposed framework eliminates the need for legal experts to interpret legal concepts . it uses large language models to extract concept-related information and interpret legal concept interpretations . |
SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | SALAD-Bench is a safety benchmark specifically designed for LLMs . it provides a robust source for evaluating both attack and defense algorithms . |
| Approach: | They propose a hierarchical safety benchmark specifically designed for LLMs . it uses a taxonomy of questions spanning three levels and a robust taxonomies based on a QA pair . |
| Outcome: | The proposed safety benchmark shows that LLMs are resilient against emerging threats and the effectiveness of contemporary defense methods. |
Fine-grained Gender Control in Machine Translation with Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work on controlled translation has only considered a simplified setup of one target gender for input. |
| Approach: | They propose a Gender-of-Entity prompting method for machine translation that takes the gender of the ambiguous entity as additional input and propose to use it to translate with correct gender inflections. |
| Outcome: | The proposed method instructs the model with fine-grained entity-level gender information to translate with correct gender inflections. |
Reinforcement Learning Based Text Style Transfer without Parallel Training Corpus (N19-1)
Copied to clipboard
| Challenge: | Existing methods for text style transfer have demonstrated considerable success, but a parallel corpus may not always be available for a transfer task. |
| Approach: | They propose a text style transfer model that uses an attention-based encoder-decoder to transfer a sentence from the source style to the target style. |
| Outcome: | The proposed model outperforms state-of-the-art methods on two different style transfer tasks. |
DRT: Deep Reasoning Translation via Long Chain-of-Thought (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown the effectiveness of long chain-of-thought (CoT) in reasoning tasks such as math and coding tasks. |
| Approach: | They propose to use Qwen2.5 and LLama-3.1 as backbones to train long thought models to bring the success of long chain-of-thought (CoT) to neural machine translation. |
| Outcome: | The proposed model outperforms vanilla LLMs and LLM models which are fine-tuning on paired sentences without long thought and outperformed vanilla LRMs. |
Paraphrase Generation with Deep Reinforcement Learning (D18-1)
Copied to clipboard
| Challenge: | Paraphrase generation is an important but challenging task in natural language processing . traditional symbolic approaches to paraphrase generation include rule-based methods, thesaurus-based approaches and statistical machine translation (SMT) |
| Approach: | They propose a deep reinforcement learning approach to automatic paraphrase generation . they propose supervised learning and reinforcement learning for evaluators . |
| Outcome: | The proposed framework outperforms state-of-the-art methods in paraphrase generation on two datasets. |
CAMIEval: Enhancing NLG Evaluation through Multidimensional Comparative Instruction-Following Analysis (2025.naacl-long)
Copied to clipboard
| Challenge: | Evaluating the quality of texts generated by language models has always been a challenging task in natural language processing (NLP). |
| Approach: | They propose a multidimensional comparative evaluation method based on instruction-following that combines relevance, factuality, and adherence with a concrete Chain-of-Thoughts process to enhance the accuracy of evaluations. |
| Outcome: | The proposed method outperforms existing methods in correlation with human evaluations on two NLG evaluation benchmarks. |
Rewriter-Evaluator Architecture for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to improve neural machine translation models with multiple decoding passes lack proper policies to terminate multi-pass processes. |
| Approach: | They propose a novel architecture of Rewriter-Evaluator to terminate multi-pass decoding . they propose prioritized gradient descent to jointly and efficiently train rewriter and evaluator . |
| Outcome: | The proposed architecture significantly outperforms existing methods on three translation tasks and reduces performance gaps to oracle policies. |
UCL-Bench: A Chinese User-Centric Legal Benchmark for Large Language Models (2025.findings-naacl)
Copied to clipboard
Ruoli Gan, Duanyu Feng, Chen Zhang, Zhihang Lin, Haochen Jia, Hao Wang, Zhenyang Cai, Lei Cui, Qianqian Xie, Jimin Huang, Benyou Wang
| Challenge: | Existing legal benchmarks focusing on knowledge and logic evaluate LLMs on various tasks in legal domain, but few have explored the practical application of LLM by actual users. |
| Approach: | They propose a Chinese user-centric legal benchmark that aims to assess the practical application of LLMs by real users. |
| Outcome: | The proposed model outperforms existing models on various tasks in legal domain but does not outperfect ChatGPT. |
SimulBench: Evaluating Language Models with Creative Simulation Tasks (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models do not fully evaluate their potential for broad implementation. |
| Approach: | They propose to use a fixed LLM as a user agent to engage with an LLM to collect dialogues first under different tasks. |
| Outcome: | The proposed framework outperforms LLaMA-3-70b-Chat on 18.55% more cases. |
Pseudointelligence: A Unifying Lens on Language Model Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies claim that language models surpass human performance on new benchmarks within a few years. |
| Approach: | They propose a framework for model evaluation that casts as a dynamic interaction between a model and a learned evaluator. |
| Outcome: | The proposed framework can be used to reason about two case studies in language model evaluation, and analyze existing evaluation methods. |
Large Language Models are not Fair Evaluators (2024.acl-long)
Copied to clipboard
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, Zhifang Sui
| Challenge: | Existing evaluation frameworks that use large language models as referees are insufficient for accurately assessing their alignment with human intent. |
| Approach: | They propose a calibration framework to address positional bias in large language models as evaluators by manually annotating the “win/tie/lose” outcomes of responses from ChatGPT and Vicuna-13B in the Vicun A Benchmark’s question prompt. |
| Outcome: | The proposed framework alleviates evaluation bias, resulting in closer alignment with human judgments. |
Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) evaluation is a patchy and inconsistent landscape . established automatic evaluation metrics are poor surrogates, correlating weakly with human judgement. |
| Approach: | They propose to use both automatic and human evaluation to evaluate generative LLMs on three NLP benchmarks: text summarisation, text simplification and grammatical error correction. |
| Outcome: | The proposed model outperforms many popular models according to human reviewers on the majority of metrics, while scoring much worse when using classic automatic evaluation metrics. |
A LLM-based Ranking Method for the Evaluation of Automatic Counter-Narrative Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for evaluating CNs are expensive, time-consuming, and subjective, but lack a universal truth and the lack of a 'universal truth' . |
| Approach: | They propose a model ranking pipeline based on pairwise comparisons of generated CNs from different models organized in a tournament-style format to improve the evaluation process. |
| Outcome: | The proposed method achieves a high correlation with human preference, with a score of 0.88, and compares chat, instruct, and base models, exploring their strengths and limitations. |
Accelerating Multiple Intent Detection and Slot Filling via Targeted Knowledge Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing non-autoregressive Spoken Language Understanding models suffer from multi-modality problem . current methods have little prior knowledge about the reference during inference . |
| Approach: | They propose a Targeted Knowledge Distillation Framework (TKDF) for multi-intent SLU that utilizes the knowledge distillation method to improve the performance. |
| Outcome: | The proposed model outperforms existing models on two public multi-intent datasets while speeding up by over 4.5 times. |
Aligning Large Language Models by On-Policy Self-Judgment (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches for aligning large language models with human preferences face a trade-off that requires a separate reward model for on-policy learning. |
| Approach: | They propose a new alignment framework that does on-policy learning and is parameter efficient . they propose Judge-augmented Supervised Fine-Tuning to train a single model to act as a policy and a judge. |
| Outcome: | The proposed framework outperforms baselines in preference benchmarks and rejecting sampling by itself improves performance without additional evaluator. |
Improving Pacing in Long-Form Story Planning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems for writing long-form stories suffer from unnatural pacing, whether glossing over important events or over-elaborating on insignificant details. |
| Approach: | They propose a system that uses a concreteness evaluator to judge which of two events is more concrete. |
| Outcome: | The proposed system improves pacing when automatically generating story outlines. |
Weak Reward Model Transforms Generative Models into Robust Causal Event Extraction Systems (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics that reflect the performance of causal event extraction tasks are poorly reflecting the inherent ambiguity of cause and effect boundaries. |
| Approach: | They propose to use a weak-to-strong supervision method to train an evaluation model while still achieving high performance in training an RL model. |
| Outcome: | The proposed method achieves high agreement with human-annotated data while still achieving high performance in training an RL model. |
EvolvR: Self-Evolving Pairwise Reasoning for Story Evaluation to Enhance Generation (2026.acl-long)
Copied to clipboard
Xinda Wang, Zhengxu Hou, Yangshijie Zhang, null Yanbingren, Jialin Liu, ChenZhuo Zhao, Zhibo Yang, Bin-Bin Yang, Feng Xiao
| Challenge: | Existing methods for story evaluation lack reasoning capabilities for open-source models . evolvR framework provides high-fidelity evaluators for story generation tasks . |
| Approach: | They propose a framework that self-synthesizes chain-of-thought data via a multi-persona strategy . they propose evolvR to provide a reward model for story generation . |
| Outcome: | The proposed framework achieves state-of-the-art performance on three evaluation benchmarks . it also enhances the quality of generated stories, validating the superiority of the framework . |
UltraIF: Advancing Instruction Following from the Wild (2025.emnlp-main)
Copied to clipboard
| Challenge: | a lack of transparency has resulted in a gap between research community and leading companies . large language models have demonstrated remarkable capabilities in following complex instructions . |
| Approach: | They propose a method to build large language models that can follow complex instructions with open-source data. |
| Outcome: | The proposed approach can synergize complex instructions and filter responses with evaluation questions. |
Argue with Me Tersely: Towards Sentence-Level Counter-Argument Generation (2023.emnlp-main)
Copied to clipboard
Jiayu Lin, Rong Ye, Meng Han, Qi Zhang, Ruofei Lai, Xinyu Zhang, Zhao Cao, Xuanjing Huang, Zhongyu Wei
| Challenge: | Existing work describes paragraph-level counter-argument generation task as paragraph-based . however, sentence-level generation can be quite different due to its unique constraints and brevity-focused challenges. |
| Approach: | They propose a benchmark framework for sentence-level counter-argument generation . they use an annotated debate forum dataset to generate high-quality counter-argments . |
| Outcome: | The proposed framework and evaluator are competitive in counter-argument generation tasks. |
Gradient-Guided Multi-Judge Prompt Optimization (2026.acl-long)
Copied to clipboard
ChenZhuo Zhao, Xinda Wang, Pu Zhao, Yue Huang, Junting Lu, Ziqian Liu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang
| Challenge: | Existing approaches to prompt optimization trade off signal quality against computational cost. |
| Approach: | They propose a framework that uses a first-order gradient approximation to score segment importance in a continuous masking direction. |
| Outcome: | The proposed framework improves efficiency and robustness by using a first-order gradient approximation to score segment importance in a continuous masking direction. |
TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignment (2026.acl-long)
Copied to clipboard
Zhewen Tan, Wenhan Yu, Jianfeng Si, Tongxin Liu, Kaiqi Guan, Huiyan Jin, Jiawen Tao, Xiaokun Yuan, Xiangzheng Zhang, Duohe Ma, Tong Yang, Lin Sun
| Challenge: | Existing approaches to safety alignment of large language models rely on costly manual annotations or human review. |
| Approach: | They propose a closed-loop reinforcement learning framework called TriPlay-RL that enables iterative collaboration among three roles with near-zero manual annotation. |
| Outcome: | The proposed framework achieves 20%–50% improvement in adversarial effectiveness while preserving high output diversity while achieving 10%–30% gains in safety performance without degrading general reasoning capability. |
Improving LLM Code Reasoning via Semantic Equivalence Self-Play with Formal Verification (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to training large language models fail to capture deep semantic properties and edge cases. |
| Approach: | They propose a framework that leverages Liquid Haskell proofs for validating equivalence and execution-based counterexamples for inequivalent training. |
| Outcome: | The proposed framework achieves 13.3pp accuracy gain on EquiBench and consistent gains on PySecDB. |
ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework (2026.acl-long)
Copied to clipboard
Kai Qin, Liangxin Liu, Yu Liang, Longzheng Wang, null Wangyan, Zhang Yueyang, Long Xia, Zhiyuan Sun, Houde Liu, Daiting Shi
| Challenge: | Existing methods for generating reward models focus on outcome-level supervision, neglecting analytical process quality, which constrains their potential. |
| Approach: | They propose a novel reward model that leverages self-reflection to assess analytical quality and enhance preference modeling. |
| Outcome: | The proposed model improves performance on four benchmarks and significantly mitigates positional bias. |