ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Evaluating personalized text generated by large language models is challenging, as only the LLM user, i.e. prompt author, can reliably assess the output. |
| Approach: | They propose an explainable reference-based evaluation framework that leverages an LLM to extract atomic aspects and their evidences from the generated and reference texts, match the aspects, and evaluate their alignment based on content and writing style. |
| Outcome: | The proposed framework achieves a 7.2% improvement in alignment with human judgments compared to the state-of-the-art evaluation methods. |
Similar Papers
SciText2Eq: Assessing LLMs for Explainable Equation Generation for Scientific Creativity (2026.findings-acl)
Copied to clipboard
| Challenge: | Prior work has addressed problems in unstructured grounding, multi-equation dependency, and human-aligned evaluation. |
| Approach: | They construct a dataset of scientific texts and evaluate it using an explainable equation generation workflow using automatic metrics and human judgments. |
| Outcome: | The proposed model achieves moderate performance on lexical and syntactic similarity, but struggles with semantic accuracy. |
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. |
| Approach: | They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them. |
| Outcome: | The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods. |
ExPerT: Personalizing LLM Responses to Users’ Domain Expertise via Query-Wise Semantic and Keystroke Behavioral Cues (2026.acl-long)
Copied to clipboard
| Challenge: | Existing personalization methods relying on static profiles or text-only signals fail to capture query-specific expertise variation. |
| Approach: | They propose a query-wise personalization framework that adapts LLM responses to query domain expertise by combining semantic and behavioral cues. |
| Outcome: | ExPerT reduces expertise inference error by 65.7% compared to the strongest baseline and improves response satisfaction by 17.52% on a 5-point Likert scale. |
Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have raised concerns about reliability and trustworthiness of the models. |
| Approach: | They analyze 134 papers and introduce a taxonomy of evidence-based text generation with LLMs. |
| Outcome: | The proposed methods highlight open challenges and outline promising directions for future work. |
Can GPT-4 Sway Experts’ Investment Decisions? (2025.findings-naacl)
Copied to clipboard
| Challenge: | In the post-Turing era, evaluating large language models involves assessing generated text based on readers’ decisions rather than merely its indistinguishability from human-produced content. |
| Approach: | They propose to use GPT-4 to evaluate generated text from the aspects of grammar, convincingness, logical coherence, and usefulness to determine its validity. |
| Outcome: | The proposed model can generate persuasive analyses affecting the decisions of amateurs and experts. |
Learning Personalized Alignment for Evaluating Open-ended Text Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Traditional evaluation metrics rely heavily on lexical similarity with human-written references, showing poor correlation with human judgments and failing to account for alignment with the diversity of human preferences. |
| Approach: | They propose an interpretable evaluation framework that evaluates alignment with specific human preferences by providing detailed comments and fine-grained scoring. |
| Outcome: | The proposed framework outperforms GPT-4 in Kendall correlation and accuracy with zero-shot reviewers. |
ReasoningRec: Bridging Personalized Recommendations and Human-Interpretable Explanations through LLM Reasoning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Empirical evaluations demonstrate that ReasoningRec surpasses state-of-the-art methods by up to 12.5% in recommendation prediction while simultaneously providing human-intelligible explanations. |
| Approach: | They propose a reasoning-based recommendation framework that leverages Large Language Models to model users and items, focusing on preferences, aversions, and explanatory reasoning. |
| Outcome: | The proposed framework surpasses state-of-the-art methods by up to 12.5% in recommendation prediction while providing human-intelligible explanations. |
Finding Blind Spots in Evaluator LLMs with Interpretable Checklists (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models are increasingly relied upon to evaluate text outputs of other LLMs . however, concerns persist over the accuracy of these assessments and the potential for misleading conclusions. |
| Approach: | They propose a framework to assess the reliability of Large Language Models (LLMs) they propose ' FBI' framework to examine the proficiency of Evaluator LLMs in assessing four critical abilities . |
| Outcome: | The proposed framework assesses the performance of LLMs in text generation tasks. |
Learning to Judge: LLMs Designing and Applying Evaluation Rubrics (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models are increasingly used as evaluators for natural language generation . human rubrics are often static and misaligned with how models internally represent language quality. |
| Approach: | They propose to use large language models to generate interpretable and task-aware evaluation dimensions and apply them within models. |
| Outcome: | The proposed model improves the semantic coherence and scoring reliability of LLM-defined criteria and their alignment with human criteria. |
Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Generative large language models generate a high-dimensional probability distribution over all tokens in their vocabulary. |
| Approach: | They conduct extensive sensitivity analyses to determine how hyperparameter choices shape the outputs of generative large language models. |
| Outcome: | The proposed methods influence the distribution of diversity and coherence metrics in human-written text, but the optimal configurations vary across models and tasks. |