Evaluating Cultural Knowledge and Reasoning in LLMs Through Persian Allusions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Allusion recognition is a critical test of LLMs' ability to deploy stored information in open-ended, figurative settings. |
| Approach: | They propose a framework for evaluating Persian literary allusions through annotations and LLM-generated texts incorporating allusion in novel contexts. |
| Outcome: | The proposed framework evaluates Persian literary allusions through annotations and LLM-generated texts incorporating allusion in novel contexts. |
Similar Papers
Evaluating the Creativity of LLMs in Persian Literary Text Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prior research has focused primarily on English, with limited exploration of non-English literary traditions and without standardized methods for assessing creativity. |
| Approach: | They build a dataset of user-generated Persian literary spanning 20 diverse topics and assess model outputs along four creativity dimensions . |
| Outcome: | The proposed models generate Persian literary text enriched with culturally relevant expressions. |
MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are used globally across many languages, but their English-centric pretraining raises concerns about cross-lingual disparities for cultural awareness . |
| Approach: | They introduce an automatic multilingual framework for evaluating cultural awareness in large language models across languages, regions, and topics. |
| Outcome: | The framework evaluates open-ended text generation, capturing how models express culturally grounded knowledge in natural language. |
Out-of-Context Reasoning in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lightweight technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
| Approach: | They propose a lightweight technique that trains only new token embeddings on axioms . they train only new embeddables and evaluate them on unseen tasks . |
| Outcome: | The proposed technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
Beyond Understanding: Evaluating the Pragmatic Gap in LLMs’ Cultural Processing of Figurative Language (2026.eacl-long)
Copied to clipboard
| Challenge: | Using figurative language as a proxy for cultural nuance and local knowledge, large language models struggle with connotative meaning. |
| Approach: | They evaluate large language models' ability to process culturally grounded language . they use figurative language as a proxy for cultural nuance and local knowledge . |
| Outcome: | The proposed models can understand and use figurative expressions that encode local knowledge and social nuance. |
Evaluating LLMs for Quotation Attribution in Literary Texts: A Case Study of LLaMa3 (2025.naacl-short)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown promising results in literary tasks . however, quotation attribution remains a challenging task and methods that generalize across writing styles are lacking analysis regarding book memorization and annotation contamination. |
| Approach: | They evaluate the ability of Llama-3 to attribute utterances of direct-speech to their speaker in novels by assessing the impact of book memorization and annotation contamination. |
| Outcome: | The proposed model outperforms existing models on a corpus of 28 novels and shows that book memorization and annotation contamination do not explain the performance gain. |
Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations (2024.acl-long)
Copied to clipboard
| Challenge: | Currently, many benchmarks evaluate the commonsense reasoning of large language models (LLMs), but most are English-based, limiting non-English evaluations. |
| Approach: | They propose to use Chinese commonsense reasoning to evaluate LLMs' commonsensing ability. |
| Outcome: | The proposed benchmark covers both globally known and Chinese-specific commonsense reasoning abilities and can be used as a reference for future research. |
Literary Evidence Retrieval via Long-Context Language Models (2025.acl-short)
Copied to clipboard
| Challenge: | a recent study shows that long-context language models can exceed human expert performance in literary analysis . despite their speed and apparent accuracy, even the strongest models struggle with nuanced literary signals and overgeneration. |
| Approach: | They propose a task where a model is given an entire text of a book and a literary criticism with a missing quotation from that work and asked to generate the missing quote. |
| Outcome: | The proposed model outperforms open-weight models in literary evidence retrieval tasks. |
PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian (2025.naacl-long)
Copied to clipboard
Erfan Moosavi Monazzah, Vahid Rahimzadeh, Yadollah Yaghoobzadeh, Azadeh Shakery, Mohammad Taher Pilehvar
| Challenge: | Large language models predominantly reflect Western cultures due to the dominance of English-centric training data. |
| Approach: | They propose a dataset to assess the sensitivity of LLMs to Persian culture. |
| Outcome: | The proposed model shows a 11.3% gap between best closed-source model and layperson baseline while the gap increases to 21.3% by using the best open-weight model. |
DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | Retrievalaugmented LLMs have been used to ground LLM in external knowledge . a gap exists in the current landscape regarding the effectiveness of grounding LLM on heterogeneous knowledge sources. |
| Approach: | They propose a model that uses symbolic language to generate symbolic queries . they use a dataset that is generated using predefined reasoning chains and human annotation . |
| Outcome: | The proposed model outperforms previous approaches by a significant margin in QA tasks over text. |
GOLEM: GOld Standard for Learning and Evaluation of Motifs (2024.lrec-main)
Copied to clipboard
| Challenge: | Motifs are distinctive, recurring, widely used idiom-like words or phrases, often originating from folklore, whose meaning are anchored in a narrative. |
| Approach: | They present a dataset annotated for motific information in English . it contains 26,078 motif candidates across 34 motif types from three cultural or national groups: Jewish, Irish, and Puerto Rican. |
| Outcome: | The first dataset annotated for motific information identifies 26,078 motif candidates across 34 motif types from three cultural or national groups: Jewish, Irish, and Puerto Rican. |