A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences (2024.emnlp-main)
Copied to clipboard
| Challenge: | syllogistic reasoning is a deductive reasoning skill that is crucial in everyday problem-solving and decision-making experiences. |
| Approach: | They propose to study the reasoning abilities of Large Language Models (LLMs) they propose to use supervised fine-tuning and chain-of-thought reasoning to investigate their results. |
| Outcome: | The proposed models exhibit reasoning biases, avoid answering that no conclusion follows, align with human difficulties, and struggle with multi-step reasoning. |
Similar Papers
Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models of large language reasoning exhibit reasoning biases similar to humans, a study shows . syllogistic reasoning is one of the basic forms of deductive reasoning . |
| Approach: | They propose to use a syllogism dataset to evaluate models' reasoning abilities . they propose to ask LLMs to translate slogismatic slurs into abstract logical expressions . |
| Outcome: | The proposed method shows that models exhibit reasoning biases similar to humans, and that there is room for improvement in reasoning problems where premises and hypotheses are neither entailment nor contradiction. |
A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Psychologists have documented several ways in which humans’ inferences deviate from the rules of logic. |
| Approach: | They focus on syllogisms, which are inferences from two simple premises, and show that larger models are more logical than smaller ones. |
| Outcome: | The results show that language models often mimic human biases, but overcome them in some cases. |
A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners (2024.emnlp-main)
Copied to clipboard
Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie Su, Camillo Taylor, Dan Roth
| Challenge: | a new hypothesis-testing framework is developed to assess whether large language models possess genuine reasoning abilities or primarily depend on token bias. |
| Approach: | They propose a framework to assess whether large language models have genuine reasoning abilities or primarily depend on token bias. |
| Outcome: | The proposed framework outlines a list of hypotheses where token biases are readily identifiable . the results suggest that most LLMs still struggle with logical reasoning . |
Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies on reasoning in language models have sparked a debate on whether they can learn systematic inferential principles or merely exploit superficial patterns in the training data. |
| Approach: | They propose a method for circuit discovery aimed at interpreting syllogistic inference . they uncover a circuit involving middle-term suppression that elucidates how LMs transfer information to derive valid conclusions from premises. |
| Outcome: | The proposed method elucidates how LMs transfer information to derive valid conclusions from premises. |
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models (2024.acl-long)
Copied to clipboard
Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, Chitta Baral
| Challenge: | Existing work investigating the logical reasoning ability of large language models has focused only on a couple of inference rules of propositional and first-order logics. |
| Approach: | They propose to use a natural language question-answering dataset to evaluate the logical reasoning ability of large language models. |
| Outcome: | The proposed model performs poorly on a range of natural language questions using chain-of-thought prompting. |
Towards Reasoning in Large Language Models: A Survey (2023.findings-acl)
Copied to clipboard
| Challenge: | Reasoning is a fundamental aspect of human intelligence that plays a crucial role in many intellectual activities. |
| Approach: | They propose to improve LLMs' ability to elicit reasoning by providing exemplars or prompts to model reasoning. |
| Outcome: | This paper provides a comprehensive overview of the state of knowledge on reasoning in large language models. |
SylloBio-NLI: Evaluating Large Language Models on Biomedical Syllogistic Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models are far from achieving the robustness and consistency required for safe biomedical NLI applications. |
| Approach: | They propose a framework that leverages external ontologies to instantiate diverse syllogistic arguments for biomedical NLI by identifying valid conclusions and extracting supporting evidence. |
| Outcome: | The proposed framework evaluates large language models on identifying valid conclusions and extracting supporting evidence across 28 syllogistic schemes instantiated with human genome pathways. |
Do Large Language Models excel in Complex Logical Reasoning with Formal Language? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on LLMs have focused on formal language, but evaluations of their performance are limited. |
| Approach: | They propose to use a formal language to evaluate LLMs across logical reasoning problems using formal languages. |
| Outcome: | The proposed model outperforms Instruct models in three dimensions, taxonomy of tasks, and format of trajectories, and achieves the best generalization performance across other languages. |
Exploring Self-supervised Logic-enhanced Training for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Traditional attempts to enhance the logical reasoning abilities of language models often rely on supervised fine-tuning, limiting their generalization to new tasks or domains. |
| Approach: | They propose a framework for integrating logical reasoning capabilities into LLMs and activating them via in-context learning. |
| Outcome: | The proposed framework achieves comparable results to existing models on three language understanding benchmarks. |
Complex Reasoning in Natural Language (2023.acl-tutorials)
Copied to clipboard
| Challenge: | Recent research shows that pretrained language models are often brittle for complex reasoning tasks. |
| Approach: | They propose to use pre-trained language models to teach machines to reason over texts . they will review recent promising approaches to tackling complex reasoning tasks . |
| Outcome: | This tutorial reviews promising approaches to complex reasoning tasks . it reviews the methods that can be used to augment models with robustness . |