Detecting (Un)answerability in Large Language Models with Linear Directions (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) often respond confidently to questions even when they lack the necessary information, leading to inaccurate responses or hallucinations. |
| Approach: | They propose an approach for identifying a direction in the model’s activation space that captures unanswerability and uses it for classification. |
| Outcome: | The proposed method detects unanswerable questions and generalizes better across datasets than existing prompt-based and classifier-based approaches. |
Similar Papers
The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been shown to possess impressive capabilities, but they are not problem-free. |
| Approach: | They explore the behavior of large language models when presented with (un)answerable queries. |
| Outcome: | The proposed models encode the answerability of an input query, the authors show . they also show that the first decoded token is a strong indicator . |
Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Abstention Ability (AA) is a critical aspect of Large Language Model (LLM) reliability. |
| Approach: | They propose a black-box evaluation approach and a new dataset, Abstain-QA, to rigorously assess AA across varied question types, domains, and task types. |
| Outcome: | The proposed evaluation process and new dataset, Abstain-QA, are crafted to rigorously assess AA across varied question types, domains, and task types. |
Leveraging LLM-Generated Schema Descriptions for Unanswerable Question Detection in Clinical Data (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods rely on model uncertainty but lack interpretability and data imbalance. |
| Approach: | They propose a lightweight model that predicts relevant database schemas to detect unanswerable questions, enhancing interpretability and addressing the data imbalance in binary classification tasks. |
| Outcome: | The proposed model improves interpretability and improves accuracy in binary classification tasks. |
Evaluating Open-Domain Question Answering in the Era of Large Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation models fail to identify lexical matching failures for open-domain question answering. |
| Approach: | They manually evaluate open-domain QA models by manually evaluating their answers on a popular benchmark. |
| Outcome: | The proposed model performs better on NQ-open than existing models and more than 50% of lexical matching failures are attributed to semantically equivalent answers. |
Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) produce incomplete or selectively omit key information . omissions of key information or misrepresentation of conflicting evidence can cause harm . |
| Approach: | They propose a method that decomposes texts into atomic statements and uses natural language inference to identify missing facts and a Q A-based metric that extracts question-answer pairs and compares responses across sources. |
| Outcome: | The proposed evaluation metrics show they perform better than more complex metrics, but at a cost. |
Constructing Interpretable Features from Compositional Neuron Groups (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for analyzing LLMs rely on dictionary learning with sparse autoencoders (SAEs) however, SAEs struggle in causal evaluations and lack intrinsic interpretability, as their learning is not explicitly tied to the computations of the model. |
| Approach: | They propose to decompose MLP activations with semi-nonnegative matrix factorization (SNMF) such that the learned features are mapped to their activating inputs, making them directly interpretable. |
| Outcome: | Experiments on Llama 3.1, Gemma 2 and GPT-2 show that SNMF derived features outperform SAEs and a strong supervised baseline on causal steering while aligning with human-interpretable concepts. |
DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | Retrievalaugmented LLMs have been used to ground LLM in external knowledge . a gap exists in the current landscape regarding the effectiveness of grounding LLM on heterogeneous knowledge sources. |
| Approach: | They propose a model that uses symbolic language to generate symbolic queries . they use a dataset that is generated using predefined reasoning chains and human annotation . |
| Outcome: | The proposed model outperforms previous approaches by a significant margin in QA tasks over text. |
Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method (2024.naacl-long)
Copied to clipboard
Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, Dawei Yin
| Challenge: | Recent literature reveals that Large Language Models (LLMs) hallucinate intermittently, which impedes their reliability for further utilization. |
| Approach: | They propose a self-detection method to detect which questions an LLM does not know by combining the two components to identify whether the model generates a non-factual response to the question. |
| Outcome: | The proposed method can detect which questions an LLM does not know across factoid question-answering, arithmetic reasoning, and commonsense reasoning tasks. |
Exploring Language Model Generalization in Low-Resource Extractive QA (2025.coling-main)
Copied to clipboard
| Challenge: | Existing LLMs struggle with dataset demands of closed domains such as medicine and law . current LLM performance in closed domain is lacking, even on traditional tasks such as Natural Language Inference . |
| Approach: | They investigate Extractive Question Answering (EQA) with Large Language Models (LLMs) under domain drift . they find that LLMs struggle with dataset demands of closed domains . |
| Outcome: | The proposed model performs poorly in extractive question answering tasks under domain drift . the proposed model can generalize to domains that require specific knowledge without training . |
Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Table Question Answering (TQA) aims to answer natural language questions using tabular data. |
| Approach: | They propose a systematic overview of TQA research using large language models and summarize available benchmarks based on task features. |
| Outcome: | The proposed framework provides a comprehensive overview of the current state of the art in the field of Table Question Answering. |