Towards a Unified Multi-Dimensional Evaluator for Text Generation

Challenge:	Existing evaluation frameworks for natural language generation are dominated by similarity-based metrics.
Approach:	They propose a multi-dimensional evaluator for natural language generation that integrates multiple dimensions into one evaluer.
Outcome:	The proposed evaluator improves on three typical NLG tasks and improves with external knowledge.

Similar Papers

Overview

Challenge

Approach

Outcome

Multi-Dimensional Evaluation of Text Summarization with In-Context Learning (2023.findings-acl)

Copied to clipboard

Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes, Pengfei Liu, Graham Neubig, Chunting Zhou

Low-Resource NLP

Challenge:	In-context learning-based evaluators are competitive with learned evaluation frameworks for text summarization tasks.
Approach:	They propose to use large language models as multi-dimensional evaluators using in-context learning to evaluate text summarization tasks.
Outcome:	The proposed frameworks are competitive with existing frameworks on relevance and factual consistency, the authors show .

QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024.emnlp-main)

Copied to clipboard

Weiping Fu, Bifan Wei, Jianxiang Hu, Zhongmin Cai, Jun Liu

Text Generation

Challenge:	Existing metrics fail to align well with human judgments when evaluating QG questions.
Approach:	They propose a multi-dimensional evaluation benchmark for QG and automatic metrics that evaluates questions and automated metrics across 7 dimensions.
Outcome:	The proposed benchmark evaluates QG models and automatic metrics across 7 dimensions . it shows that most QG model performs unsatisfactorily in terms of answerability and answer consistency .

CAMIEval: Enhancing NLG Evaluation through Multidimensional Comparative Instruction-Following Analysis (2025.naacl-long)

Copied to clipboard

Ziyue Fan, Junliang He, Li Xiaoqing, Shaohui Kuang, Kai Song, Yaqian Zhou, Xipeng Qiu

Text Generation

Challenge:	Evaluating the quality of texts generated by language models has always been a challenging task in natural language processing (NLP).
Approach:	They propose a multidimensional comparative evaluation method based on instruction-following that combines relevance, factuality, and adherence with a concrete Chain-of-Thoughts process to enhance the accuracy of evaluations.
Outcome:	The proposed method outperforms existing methods in correlation with human evaluations on two NLG evaluation benchmarks.

G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment (2023.emnlp-main)

Copied to clipboard

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, Chenguang Zhu

Text Generation

Challenge:	Conventional reference-based metrics have low correlation with human judgments, especially for open-ended generation tasks.
Approach:	They propose to use large language models as reference-free NLG evaluators to assess the quality of NLG outputs.
Outcome:	The proposed framework outperforms all previous methods in two generation tasks, and has a Spearman correlation of 0.514 with human on summarization task, and a large variance in human judgments.

X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects (2024.naacl-long)

Copied to clipboard

Minqian Liu, Ying Shen, Zhiyang Xu, Yixin Cao, Eunah Cho, Vaibhav Kumar, Reza Ghanadan, Lifu Huang

Text Generation

Challenge:	X-Eval is a two-stage instruction tuning framework to evaluate text in both seen and unseen aspects customized by end users.
Approach:	They introduce a two-stage instruction tuning framework to evaluate text in both seen and unseen aspects customized by end users.
Outcome:	The proposed framework improves the model’s ability to follow evaluation instructions and enhances the learning stage to better assess text quality.

Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation (2026.acl-long)

Copied to clipboard

Jiaju Chen, Yuxuan Lu, Xiaojie Wang, Huimin Zeng, Jing Huang, Jiri Gesi, Ying Xu, Bingsheng Yao, Dakuo Wang

Challenge:	Existing "LLM-as-a-judge" evaluation frameworks are limited by persona descriptions and are not generalizable to other tasks.
Approach:	They propose a framework that can automatically construct multiple evaluator personas with distinct dimensions from relevant text documents and instantiate LLM agents with the persona.
Outcome:	The proposed framework can believably simulate human evaluators . it extracts stakeholders' diverse perspectives from the provided research papers and constructs personas for the agents .

Unveiling the Achilles’ Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models (2024.findings-acl)

Copied to clipboard

Yiming Chen, Chen Zhang, Danqing Luo, Luis Fernando D’Haro, Robby Tan, Haizhou Li

Robustness in NLP

Challenge:	Recent studies have highlighted various neural metrics that align well with human evaluations.
Approach:	They propose a black-box adversarial framework that generates strong disagreements between human and victim evaluators.
Outcome:	The proposed framework can significantly improve the performance of human and victim evaluators.

A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)

Copied to clipboard

Mitesh M. Khapra, Ananya B. Sai

Text Generation

Challenge:	This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field.
Approach:	This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement .
Outcome:	This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field.

UniSumEval: Towards Unified, Fine-grained, Multi-dimensional Summarization Evaluation for LLMs (2024.findings-emnlp)

Copied to clipboard

Yuho Lee, Taewon Yun, Jason Cai, Hang Su, Hwanjun Song

Summarization

Challenge:	Existing benchmarks for summarization quality evaluation lack diverse input scenarios, focus on narrowly defined dimensions, and struggle with subjective and coarse-grained annotation schemes.
Approach:	They propose to use AI to help human annotations and identifie potentially hallucinogenic input texts.
Outcome:	The proposed benchmarks improve on existing benchmarks in terms of input diversity, granularity of human annotations, and evaluation dimensions.

DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering (2023.acl-long)

Copied to clipboard

Pei Ke, Fei Huang, Fei Mi, Yasheng Wang, Qun Liu, Xiaoyan Zhu, Minlie Huang

Responsible & Trustworthy NLP

Challenge:	Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability.
Approach:	They propose a metric that evaluates natural language generation tasks as an instruction-style question answering task and utilizes instruction-tuned pre-trained language models without training on evaluation datasets.
Outcome:	The proposed metric achieves state-of-the-art performance in untrained metrics for evaluating text summarization and dialogue generation, which exhibits strong dimension-level / task-level generalization ability and interpretability.