Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can’t Answer? (2025.naacl-short)
Copied to clipboard
| Challenge: | Question answering (QA) is a popular task, but we test both separately . a recent study found that LLMs are less accurate in numerical RQA than RQA . |
| Approach: | We run 16 LLMs on QA and RQA with trivia questions/answers . they find question and answer types that lead to RQA errors and suggest improvements . |
| Outcome: | The results show that LLMs are less accurate in RQA for numerical answers than RQA . RQA errors correlate with question difficulty and inversely correlate with answer frequencies . |
Similar Papers
Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above (2025.acl-long)
Copied to clipboard
| Challenge: | Multiple choice question answering (MCQA) is popular for LLM evaluation due to its simplicity and human-like testing. |
| Approach: | They argue for a reform of multiple choice question answering (MCQA) they argue for more generative formats based on human testing . |
| Outcome: | The proposed reforms improve the quality of MCQA, the authors argue . they show that even when MCQ is a useful format, its datasets suffer from leakage, unanswerability, shortcuts and saturation. |
Bend but Don’t Break? Multi-Challenge Stress Test for QA Models (D19-58)
Copied to clipboard
| Challenge: | a gap remains in reasoning ability compared to a human, and performance tends to degrade when models are exposed to less-constrained tasks. |
| Approach: | They conduct extensive qualitative and quantitative analyses on the results of four models across four datasets . they relate common errors to model capabilities and discuss a way forward . |
| Outcome: | The proposed model performance is based on the results of four models across four datasets. |
Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models have been shown to have worse abstention abilities than reasoning models . a new class of abstraction methods is developed to improve absttention performance . |
| Approach: | They propose a class of abstention methods that generate reasoning trace and reconstruct most likely query from it. |
| Outcome: | The proposed method beats baselines in 33 out of 36 settings. |
PEDANTS: Cheap but Effective and Interpretable Answer Equivalence (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current short-form QA evaluations lack diverse styles of evaluation data and rely on expensive and slow LLMs. |
| Approach: | They propose a rubric for machine QA that is more stable than an exact match and neural methods. |
| Outcome: | The proposed evaluations improve on the existing short-form QA evaluations using the Trivia community. |
ARC ‘Challenge’ Is Not That Challenging (2025.findings-acl)
Copied to clipboard
| Challenge: | ARC Challenge appears to be more difficult than ARC Easy for modern LLMs due to an evaluation setup that prevents direct comparison of answer choices rather than inherent complexity. |
| Approach: | They propose a setup where multiple choice problems are evaluated and the one with the highest likelihood is compared against the gold standard to determine accuracy. |
| Outcome: | The proposed evaluation setup is more difficult than ARC Easy for modern LLMs because it prevents direct comparison of answer choices rather than inherent complexity. |
Knowledge Corpus Error in Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in open-domain question answering (QA) has explored generating context passages from large language models (LLMs) however, it is not well understood why generated passages can be more effective than retrieved ones. |
| Approach: | They propose to generate context passages from large language models by paraphrasing human-annotated gold context using LLMs to observe knowledge corpus error. |
| Outcome: | The proposed framework shows that paraphrasing human-annotated gold contexts improves performance over retrieval steps. |
NoiseQA: Challenge Set Evaluation for User-Centric Question Answering (2021.eacl-main)
Copied to clipboard
| Challenge: | Question-Answering (QA) systems are deployed in the real world . a lack of research attention has been devoted to studying the issues that arise when people use QA systems. |
| Approach: | They show that component components that precede an answering engine can introduce varied and considerable sources of error. |
| Outcome: | The proposed evaluations highlight the need for QA evaluation to expand to consider real-world use. |
Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Table Question Answering (TQA) aims to answer natural language questions using tabular data. |
| Approach: | They propose a systematic overview of TQA research using large language models and summarize available benchmarks based on task features. |
| Outcome: | The proposed framework provides a comprehensive overview of the current state of the art in the field of Table Question Answering. |
I Could’ve Asked That: Reformulating Unanswerable Questions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language models do not assist users in reformulating unanswerable questions . a recent study found that the models failed to reformulate questions based on assumptions that conflict with or cannot be verified with the information available in documents. |
| Approach: | They evaluate open-source and proprietary LLMs on couldAsk to evaluate their performance . they found that GPT-4 and Llama2-7B successfully reformulate questions only 26% and 12% of the time . |
| Outcome: | The proposed model successfully reformulates questions only 26% and 12% of the time . the proposed model is not able to reformulate questions, but it can be improved . |
Can LLMs Learn From Mistakes? An Empirical Study on Reasoning Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work has shown that simple learning can enhance the chain-of-thought (CoT) reasoning of large language models. |
| Approach: | They construct mistake-correction datasets to identify and correct mistakes in CoTs . they conclude that LLMs can learn from mistakes to enhance their CoT reasoning . |
| Outcome: | The proposed datasets show that LLMs can learn from mistakes to enhance their CoT reasoning performance. |