Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have catalyzed numerous AI applications, among which role-playing agents (RPAs) are particularly popular. |
| Approach: | They propose to evaluate LLMs' character understanding capability via the character profiling task, i.e., summarizing character profiles from corresponding materials, a widely adopted yet understudied practice for RPA development. |
| Outcome: | The proposed model outperforms existing models and literature summarization methods and proves its ability to understand fictional characters in downstream tasks. |
Similar Papers
Characteristic AI Agents via Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Commercial products have been devoted to creating character-driven chatbots using large language models, but academic research in this area remains relatively scarce. |
| Approach: | They investigate the performance of LLMs in constructing characteristic AI agents by simulating real-life individuals across different settings. |
| Outcome: | The proposed benchmark compared LLMs with real-life individuals in different settings and includes evaluation metrics. |
Character is Destiny: Can Persona-assigned Language Models Make Personal Choices? (2025.findings-emnlp)
Copied to clipboard
Rui Xu, Xintao Wang, Jiangjie Chen, Siyu Yuan, Xinfeng Yuan, Jiaqing Liang, Zulong Chen, null Xiaoqingdong, Yanghua Xiao
| Challenge: | Recent research has demonstrated the potential of using LLMs to develop role-playing language agents (RPLAs) however, imitative decision-making necessitates a more nuanced understanding of personas. |
| Approach: | They propose a method that uses persona-based memory retrieval to improve RPLAs. |
| Outcome: | The proposed method significantly advances RPLAs on this task. |
Evaluating Large Language Models via Linguistic Profiling (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) undergo extensive evaluation against various benchmarks collected in established leaderboards to assess their performance across multiple tasks. |
| Approach: | They propose a new evaluation methodology to test LLMs' sentence generation abilities under specific linguistic constraints. |
| Outcome: | The proposed evaluation methodology is based on the 'linguistic profiling' approach and is not intended to be a task-oriented evaluation. |
CharacterCraft: Bridging the Literature-Reality Dialogue Gap for Practical Role-Playing Agents (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing dialogue datasets have a bias between query distributions and real-world user language usage. |
| Approach: | They propose a framework for Chinese role-playing and a robust evaluation method . they propose specialized Chinese dialogue extraction model and specialized memory retrieval module . |
| Outcome: | The proposed framework extracts character dialogue from novels and ensures high data quality. |
Evaluating LLMs for Quotation Attribution in Literary Texts: A Case Study of LLaMa3 (2025.naacl-short)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown promising results in literary tasks . however, quotation attribution remains a challenging task and methods that generalize across writing styles are lacking analysis regarding book memorization and annotation contamination. |
| Approach: | They evaluate the ability of Llama-3 to attribute utterances of direct-speech to their speaker in novels by assessing the impact of book memorization and annotation contamination. |
| Outcome: | The proposed model outperforms existing models on a corpus of 28 novels and shows that book memorization and annotation contamination do not explain the performance gain. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
Assessing the Capabilities of Large Language Models in Coreference: An Evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are a new approach to coreference resolution, but their performance is not yet fully understood. |
| Approach: | They propose that future efforts should improve scope, data, and evaluation methods of traditional coreference research to adapt to the development of LLMs. |
| Outcome: | The proposed methods improve scope, data, and evaluation methods of traditional coreference research to adapt to the development of LLMs. |
Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmark (2024.emnlp-main)
Copied to clipboard
Elizabeth Fons, Rachneet Kaur, Soham Palande, Zhen Zeng, Tucker Balch, Manuela Veloso, Svitlana Vyetrenko
| Challenge: | Large Language Models (LLMs) are a critical tool for time series analysis and reporting in many fields, including healthcare, finance, climate, and many more. |
| Approach: | They propose a framework for rigorously evaluating the capabilities of Large Language Models (LLMs) on time series understanding, encompassing both univariate and multivariate forms. |
| Outcome: | The proposed framework delineates various characteristics inherent in time series data. |
Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization (2024.findings-naacl)
Copied to clipboard
Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, Arman Cohan
| Challenge: | Recent studies have found that large language models (LLMs) can achieve state-of-the-art performance on generic summarization benchmarks, but their performance on more complex summarizing task settings is less studied. |
| Approach: | They benchmark large language models on instruction controllable text summarization . they use 4 evaluation protocols and 11 LLMs to evaluate their performance . |
| Outcome: | The proposed model performs well on instruction controllable text summarization tasks with 4 evaluation protocols and 11 LLMs. |
On Evaluating LLMs’ Capabilities as Functional Approximators: A Bayesian Evaluation Framework (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized the way we can formulate tasks in text-in-text-out format. |
| Approach: | They propose a new evaluation framework to comprehensively assess LLMs’ function modeling abilities by adopting a Bayesian perspective of function modeling. |
| Outcome: | The proposed evaluation framework enables LLMs to excel in utilizing prior knowledge to develop a strong understanding of the underlying function. |