Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation of Non-Literal Intent Resolution in LLMs (2024.acl-short)
Copied to clipboard
| Challenge: | Existing work on discriminative evaluations of large language models has focused on discrimination, but this paper examines their intention understanding by examining their responses to non-literal utterances. |
| Approach: | They propose a framework to evaluate large language models’ intention understanding by examining their responses to non-literal utterances. |
| Outcome: | The proposed framework compares large language models' responses to human-like expectations and provides nuanced evaluations of their intention understanding. |
Similar Papers
Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies have focused mainly on LLMs' comprehension of verbal behavior, with non-verbal behavior considered only in conjunction with verbal responses. |
| Approach: | They present the first systematic evaluation of LLMs’ ability to infer pragmatic meaning in dialogue consisting solely of non-verbal responses. |
| Outcome: | The proposed model fails to capture non-verbal intent and has accuracy dropping by 60% compared to verbal ones. |
“I understand your perspective”: LLM Persuasion through the Lens of Communicative Action Theory (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can generate high-quality arguments, yet their ability to engage in nuanced and persuasive communicative actions remains largely unexplored. |
| Approach: | They examine whether Large Language Models express illocutionary intent in ways comparable to human communication by simulated online discussions . |
| Outcome: | The proposed models express illocutionary intents in ways comparable to human communication, and crowd-sourced workers prefer them over human-written ones. |
LLM-based Literal Example Generation for Japanese Multiword Expressions (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing work on Japanese multiword expressions has focused on detecting idiomatic usages in context, leaving literal readings underrepresented. |
| Approach: | They propose to use corpus non-literal usages as contrastive cues for controlled prompting . they compare their results to a test that compares model predictions with human judgments . |
| Outcome: | The proposed model provides more accurate literal examples than prompts that include no hints or literal information. |
On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) provide a data-centric solution to alleviate limitations of real-world data with synthetic data generation. |
| Approach: | They propose a generic workflow for LLM-driven synthetic data generation. |
| Outcome: | The proposed workflows highlight gaps in existing research and outline avenues for future studies. |
DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | Retrievalaugmented LLMs have been used to ground LLM in external knowledge . a gap exists in the current landscape regarding the effectiveness of grounding LLM on heterogeneous knowledge sources. |
| Approach: | They propose a model that uses symbolic language to generate symbolic queries . they use a dataset that is generated using predefined reasoning chains and human annotation . |
| Outcome: | The proposed model outperforms previous approaches by a significant margin in QA tasks over text. |
Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have advanced machine translation (MT) a meta-evaluation dataset focused on non-literal translations is lacking . experimental results show the inaccuracies of traditional MT metrics and the limitations of LLM-as-a-Judge. |
| Approach: | They propose a meta-evaluation framework that leverages sub-agents to evaluate machine translation metrics. |
| Outcome: | The proposed framework improves on the knowledge cutoff and score inconsistency problem. |
They want to pretend not to understand: The Limits of Current LLMs in Interpreting Implicit Content of Political Discourse (2025.findings-acl)
Copied to clipboard
| Challenge: | a recent study shows that large language models lack the pragmatic capabilities needed to interpret highly implicit content. |
| Approach: | They propose to use transcribed italian political speeches to test their ability to interpret implicit content. |
| Outcome: | The proposed model provides a fully correct explanation in only one-fourth of cases in the open-ended generation setup. |
Factuality of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Das, Preslav Nakov
| Challenge: | Large language models (LLMs) are factually incorrect, which limits their applicability in real-world scenarios. |
| Approach: | They analyze existing work to identify major challenges and their associated causes . they propose to evaluate LLMs using a variety of measures to mitigate factual errors . |
| Outcome: | The proposed methods are based on a variety of datasets and proposed strategies to mitigate factual errors. |
Pragmatic inference of scalar implicature by LLMs (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing Large Language Models (LLMs) engage in pragmatic inference of scalar implicature, such as some. |
| Approach: | They investigate how Large Language Models (LLMs) engage in pragmatic inference of scalar implicature, such as some. |
| Outcome: | The proposed models interpret some as pragmatic implicature not all in the absence of context, aligning with human language processing. |
Generative Interfaces for Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models are increasingly seen as assistants, copilots, and consultants . however, their linear request-response format often makes interactions inefficient in multi-turn tasks . |
| Approach: | They propose a paradigm in which large language models respond to user queries by generating user interfaces that enable more adaptive and interactive engagement. |
| Outcome: | The proposed paradigm outperforms traditional chat-based interfaces in many tasks and interaction patterns. |