A Unified Agentic Framework for Evaluating Conditional Image Generation (2025.acl-long)
Copied to clipboard
Jifang Wang, Yangxue Yangxue, Longyue Wang, Zhenran Xu, Yiyu Wang, Yaowei Wang, Weihua Luo, Kaifu Zhang, Baotian Hu, Min Zhang
| Challenge: | Conditional image generation is a popular and personalization-oriented task, but there are challenges in developing task-agnostic, reliable, and explainable evaluation metrics. |
| Approach: | They propose a unified agentic framework for comprehensive evaluation of conditional image generation tasks. |
| Outcome: | The proposed framework achieves a high correlation with human assessments on seven prominent image generation tasks. |
Similar Papers
VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing metrics for conditional image generation are opaque and lack explainability . evaluators of these metrics have limited ability to evaluate image synthesis tasks . |
| Approach: | They propose a Visual Instruction-guided Explainable metric for evaluating conditional image models. |
| Outcome: | The proposed model achieves a high Spearman correlation with human evaluations, but is weaker than GPT-4o and GPT-v in evaluating synthetic images. |
Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark (2025.acl-long)
Copied to clipboard
| Challenge: | Existing MLLMs rely on commercial models such as GPT-4o for evaluations, but they are not universally accessible. |
| Approach: | They propose a task decomposition evaluation framework based on GPT-4o to automatically construct a specialized training dataset to break down the multifaceted evaluation process into simpler sub-tasks. |
| Outcome: | The proposed framework outperforms the current state-of-the-art GPT-4o evaluation framework with over 4.6% improvement in Spearman and Kendall correlations with human judgments. |
UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion (2024.acl-long)
Copied to clipboard
| Challenge: | Existing text-to-image diffusion models generate images from text prompts due to inherent brevity of textual descriptions . however, the ability to accurately synthesize images with intricate details, such as specific entities or scenes, is limited due to the inherent bribery of text descriptions. |
| Approach: | They propose a multimodal conditional diffusion framework that operates on multimodal prompts with interleaved textual and visual inputs. |
| Outcome: | The proposed framework excels in both text-to-image generation and zero-shot subject-driven synthesis. |
MSVBench: Towards Human-Level Evaluation of Multi-Shot Video Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods for complex multi-shot video are anchored to single-shot paradigms, lacking comprehensive story assets and cross-shot metrics. |
| Approach: | They propose a framework that synergizes the high-level semantic reasoning of Large Multimodal Models with the fine-grained perceptual rigor of domain-specific expert models. |
| Outcome: | The proposed framework synergizes the high-level semantic reasoning of Large Multimodal Models with the fine-grained perceptual rigor of domain-specific expert models. |
Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods rely on rigid pipelines that overlook user needs and provide numerical results without clear explanations. |
| Approach: | They propose an evaluation framework that employs human-like strategies for efficient, dynamic, multi-round evaluations using only a few samples per round. |
| Outcome: | The evaluation agent framework reduces evaluation time to 10% of traditional methods while delivering comparable results. |
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022.emnlp-main)
Copied to clipboard
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, Jiawei Han
| Challenge: | Existing evaluation frameworks for natural language generation are dominated by similarity-based metrics. |
| Approach: | They propose a multi-dimensional evaluator for natural language generation that integrates multiple dimensions into one evaluer. |
| Outcome: | The proposed evaluator improves on three typical NLG tasks and improves with external knowledge. |
GenPilot: A Multi-Agent System for Test-Time Prompt Optimization in Image Generation (2025.findings-emnlp)
Copied to clipboard
Wen Ye, Zhaocheng Liu, Gui Yuwei, Tingyu Yuan, Yunyue Su, Bowen Fang, Chaoyang Zhao, Qiang Liu, Liang Wang
| Challenge: | Existing methods for text-to-image synthesis lack systematic error analysis and refinement strategies, resulting in limited reliability and effectiveness. |
| Approach: | They propose a plug-and-play multi-agent system called GenPilot that integrates error analysis, clustering-based adaptive exploration, fine-grained verification and a memory module for iterative optimization. |
| Outcome: | The proposed method improves text consistency and structural coherence on images with a plug-and-play system. |
QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing metrics fail to align well with human judgments when evaluating QG questions. |
| Approach: | They propose a multi-dimensional evaluation benchmark for QG and automatic metrics that evaluates questions and automated metrics across 7 dimensions. |
| Outcome: | The proposed benchmark evaluates QG models and automatic metrics across 7 dimensions . it shows that most QG model performs unsatisfactorily in terms of answerability and answer consistency . |
GPTScore: Evaluate as You Desire (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluation frameworks for text generation are not adequate to assess the quality of the generated outputs. |
| Approach: | They propose a framework that utilizes emergent abilities of generative pre-trained models to evaluate generated texts. |
| Outcome: | The proposed evaluation framework can achieve what one desires to evaluate for texts simply by natural language instructions. |
HypoEval: Hypothesis-Guided Evaluation for Natural Language Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing frameworks for LLM-as-a-judge use zero-shot setting without consulting any human input, which leads to low alignment, or fine-tune LLMs on labeled data, which requires a non-trivial number of samples. |
| Approach: | They propose a hypothesis-guided evaluation framework that uses a small corpus of human evaluations to generate more detailed rubrics for human judgments and incorporates a checklist-like approach to combine LLM’s assigned scores on each decomposed dimension to acquire overall scores. |
| Outcome: | The proposed framework outperforms existing frameworks in both human rankings and human scores with 30 human evaluations and fine-tunes LLMs on labeled data with 3 times more human evaluation by 11.95%. |