Lexical Recall or Logical Reasoning: Probing the Limits of Reasoning Abilities in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing work on LLMs assesses logic abilities independently from lexical memory. |
| Approach: | They propose to assess LLMs' logic abilities independently from lexical memory . they use two sets of grid puzzles in 42 different sizes and 12 difficulty levels . |
| Outcome: | The proposed benchmarks show that LLMs are limited to a few steps of reasoning . the results show that the applied obfuscation strategies help mitigate effects of logic puzzles being part of training data. |
Similar Papers
Puzzle Solving using Reasoning of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated their logical reasoning abilities across various domains. |
| Approach: | They propose to divide puzzles into rule-based and rule-less categories and critically assess LLMs' performance through various methodologies. |
| Outcome: | The proposed models have demonstrated capabilities in deductive reasoning and inductive reasoning, but they face limitations in inductive thinking. |
Working Memory Identifies Reasoning Limits in Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using large language models, we examine the limitations of their cognitive capabilities and their working memory. |
| Approach: | They examine the limitations of large language models from a scaling perspective . they also assess various prompting strategies, revealing their diverse impacts on LLM performance. |
| Outcome: | The proposed models perform poorly on n-back tasks and on prompting strategies. |
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models (2024.acl-long)
Copied to clipboard
Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral
| Challenge: | Existing work investigating the logical reasoning ability of large language models has focused only on a couple of inference rules of propositional and first-order logics. |
| Approach: | They propose to use a natural language question-answering dataset to evaluate the logical reasoning ability of large language models. |
| Outcome: | The proposed model performs poorly on a range of natural language questions using chain-of-thought prompting. |
Towards Reasoning in Large Language Models: A Survey (2023.findings-acl)
Copied to clipboard
| Challenge: | Reasoning is a fundamental aspect of human intelligence that plays a crucial role in many intellectual activities. |
| Approach: | They propose to improve LLMs' ability to elicit reasoning by providing exemplars or prompts to model reasoning. |
| Outcome: | This paper provides a comprehensive overview of the state of knowledge on reasoning in large language models. |
Do Large Language Models excel in Complex Logical Reasoning with Formal Language? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on LLMs have focused on formal language, but evaluations of their performance are limited. |
| Approach: | They propose to use a formal language to evaluate LLMs across logical reasoning problems using formal languages. |
| Outcome: | The proposed model outperforms Instruct models in three dimensions, taxonomy of tasks, and format of trajectories, and achieves the best generalization performance across other languages. |
LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models (2024.emnlp-main)
Copied to clipboard
Yuxuan Wan, Wenxuan Wang, Yiliu Yang, Youliang Yuan, Jen-tse Huang, Pinjia He, Wenxiang Jiao, Michael Lyu
| Challenge: | LogicAsker examines and improves the reasoning abilities of large language models such as ChatGPT and GPT-4. |
| Approach: | They propose a set of atomic reasoning skills grounded in propositional and predicate logic to examine and improve the reasoning abilities of large language models such as ChatGPT and GPT-4. |
| Outcome: | The proposed approach improves reasoning abilities in large language models such as ChatGPT and GPT-4 by up to 5%. |
Conditional and Modal Reasoning in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Inferences involving conditionals and modals play a central role in the fundamental human ability to reason about distal possibilities. |
| Approach: | They propose to assess the extent to which 29 LLMs are able to distinguish logically correct inferences from logical fallacious ones. |
| Outcome: | The LLMs that make basic mistakes with conditionals and modals display inconsistent judgments across inference patterns involving epistemic modal and conditionals, and give answers about complex conditional inferences that do not match reported human judgments. |
Evaluating the Deductive Competence of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing large language models have limited abilities to solve deductive reasoning problems . performance differences between conditions do not improve overall performance . |
| Approach: | They investigate whether several large language models can solve a deductive reasoning problem in their conventional form. |
| Outcome: | The proposed models can solve a classic type of deductive reasoning problem in their conventional form. |
CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challenge (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities, but their capabilities in cryptographic decryption tasks remain underexplored. |
| Approach: | They propose a benchmark to evaluate the reasoning capabilities of large language models in cryptographic decryption tasks. |
| Outcome: | The proposed benchmark examines the reasoning capabilities of large language models in cryptographic decryption tasks. |
Logic Haystacks: Probing LLMs’ Long-Context Logical Reasoning (Without Easily Identifiable Unrelated Padding) (2026.eacl-short)
Copied to clipboard
| Challenge: | Recent large language models claim long context windows, but evaluations often involve simple retrieval tasks or synthetic tasks padded with irrelevant text. |
| Approach: | They use grammars to generate simplified English with logical representations to create long input text while controlling its semantics. |
| Outcome: | The proposed model performs better with realistic distractors than with standard models. |