A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Tasks (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Using a zero-shot classification model, we extracted multi-dimensional evaluative features from human expert critiques and used them to evaluate selected VLMs such as Llama, Qwen, or Gemini. |
| Approach: | They constructed a quantitative framework for Chinese painting critique by extracting multi-dimensional evaluative features from human expert critiques using a zero-shot classification model. |
| Outcome: | The framework was constructed by extracting features from human critiques using a zero-shot classification model. |
Similar Papers
Capabilities and Evaluation Biases of Large Language Models in Classical Chinese Poetry Generation: A Case Study on Tang Poetry (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly applied to creative domains, yet performance in classical Chinese poetry generation and evaluation remains poorly understood. |
| Approach: | They propose a framework that combines computational metrics, LLM-as-a-judge assessment, and human expert validation to evaluate large language models. |
| Outcome: | The proposed framework evaluates state-of-the-art LLMs across multiple dimensions of poetic quality in Tang poetry generation. |
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. |
| Approach: | They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them. |
| Outcome: | The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods. |
ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | In this position paper, we argue that human evaluation of generative large language models (LLMs) should be a multidisciplinary undertaking that draws upon the insights from disciplines such as user experience research and human behavioral psychology to ensure that the results are reliable. |
| Approach: | They propose a framework for human evaluation of generative large language models that takes into account usability, aesthetics and cognitive biases. |
| Outcome: | The proposed framework is based on the framework proposed by Deutsch and alnajjar . it is aimed at ensuring that human evaluation is accurate in the age of generative AI . |
IRR: Image Review Ranking Framework for Evaluating Vision-Language Models (2025.coling-main)
Copied to clipboard
Kazuki Hayashi, Kazuma Onishi, Toma Suzuki, Yusuke Ide, Seiji Gobara, Shigeki Saito, Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe
| Challenge: | Large-scale vision language models excel at generating factual content, but their ability to rank images from multiple perspectives has not been explored. |
| Approach: | They propose a framework to evaluate large-scale vision-language models by measuring their ability to rank image texts from multiple perspectives. |
| Outcome: | The proposed evaluation framework measures how closely LVLMs' judgments align with human interpretations. |
Towards A “Novel” Benchmark: Evaluating Literary Fiction with Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) context windows have enabled them to process inputs over 100K tokens and generate outputs of up to 10K token. |
| Approach: | They propose a multi-level evaluation framework that incorporates ten metrics across the Macro, Meso, and Micro levels and an annotated fiction dataset. |
| Outcome: | The proposed framework incorporates ten metrics across the Macro, Meso, and Micro levels and is based on a human-human-AI dataset. |
Modeling, Evaluating, and Embodying Personality in LLMs: A Survey (2025.findings-emnlp)
Copied to clipboard
Iago Alves Brito, Julia Soares Dollis, Fernanda Bufon Färber, Pedro Schindler Freire Brasil Ribeiro, Rafael Teixeira Sousa, Arlindo Rodrigues Galvão Filho
| Challenge: | This survey provides a comprehensive overview of the LLM-driven personality scenario. |
| Approach: | This survey provides a comprehensive overview of the LLM-driven personality scenario. |
| Outcome: | The proposed taxonomy analyzes the limitations of existing methods and identifies key research gaps. |
LLMCrit: Teaching Large Language Models to Use Criteria (2024.findings-acl)
Copied to clipboard
| Challenge: | Current research on using criteria to provide feedback on tasks is limited . a general framework that can be used to teach large language models to use criteria is lacking . |
| Approach: | They propose a framework that enables large language models to use criteria for feedback . criteria are extracted from guidelines and construct in-context demonstrations for each criterion . |
| Outcome: | The proposed framework can be used to provide natural language feedback on tasks. |
Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework (2025.findings-acl)
Copied to clipboard
Kaishuai Xu, Tiezheng Yu, Yi Cheng, Wenjun Hou, Liangyou Li, Xin Jiang, Lifeng Shang, Qun Liu, Wenjie Li
| Challenge: | Existing methods for fine-tuning open-source LLMs are limited to text-based analysis under predefined general criteria. |
| Approach: | They propose a framework that fine-tunes LLMs to replicate the evaluation explanations and judgments of proprietary models. |
| Outcome: | The proposed evaluation framework outperforms existing fine-tuned evaluation methods in effectiveness and robustness. |
CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation (2024.acl-long)
Copied to clipboard
Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, Minlie Huang
| Challenge: | Existing models for NLP evaluations lack the ability to generate informative critiques in pointwise grading and pairwise comparison especially without references. |
| Approach: | They propose a method which can acquire pointwise grading critiques with pseudo references and revise these critiques via multi-path prompting to obtain informative evaluation data in different tasks and settings. |
| Outcome: | The proposed method outperforms all open-source models and even GPT-4 in system-level correlations of pointwise grading. |
V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Models (2026.findings-acl)
Copied to clipboard
Xiangxi Zheng, Linjie Li, Zhengyuan Yang, Ping Yu, Alex Jinpeng Wang, Rui Yan, Yuan Yao, Lijuan Wang
| Challenge: | Existing static image-text benchmarks are insufficient for evaluating multimodal large language models’ dynamic perception and interactive reasoning abilities. |
| Approach: | They propose a game-based evaluation framework to assess multimodal large language models’ visual reasoning in dynamic, continuous-space environments. |
| Outcome: | The proposed framework systematically assesses MLLMs’ visual reasoning in dynamic, continuous-space environments. |