Unveiling Internal Reasoning Modes in LLMs: A Deep Dive into Latent Reasoning vs. Factual Shortcuts with Attribute Rate Ratio (2025.emnlp-main)
Copied to clipboard
Yiran Yang, Haifeng Sun, Jingyu Wang, Qi Qi, Zirui Zhuang, Huazheng Wang, Pengfei Ren, Jing Wang, Jianxin Liao
| Challenge: | Existing research in multi-hop questions has identified two reasoning modes, but has not investigated how these modes differ during inference. |
| Approach: | They propose a classification metric that compares latent reasoning and factual shortcuts in multi-hop questions. |
| Outcome: | The proposed metric achieves 90% accuracy on the proposed datasets and demonstrates effectiveness in RAG conflict scenarios. |
Similar Papers
Investigating Multi-Hop Factual Shortcuts in Knowledge Editing of Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent work has demonstrated the power of large language models in recalling knowledge and reasoning. |
| Approach: | They propose to erase shortcut neurons to mitigate the associated risks . 20% of the failures are attributed to shortcuts, they find . |
| Outcome: | The proposed approach reduces failures in multi-hop knowledge editing caused by shortcuts by 20% . |
Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts? (2025.findings-acl)
Copied to clipboard
| Challenge: | Latent multi-hop reasoning is a problem in Large Language Models that can develop shortcuts by encountering the head entity and answer entity in training sequences. |
| Approach: | They propose desiderata for shortcut-free evaluation of latent multi-hop reasoning ability . they exclude test queries where head and answer entities might have co-appeared . |
| Outcome: | The proposed model can latently recall and compose single-hop facts without shortcuts, but only for certain types of queries. |
Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers? (2024.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art Large Language Models (LLMs) are accredited with a number of different capabilities, including reading comprehension, mathematical and reasoning skills, and possessing scientific knowledge. |
| Approach: | They propose a benchmark to generate seemingly plausible multi-hop reasoning chains that ultimately lead to incorrect answers. |
| Outcome: | The proposed model circumvents the reasoning requirement but in subtle ways . it shows that it is more difficult to generate plausible alternatives . |
Do Large Language Models Latently Perform Multi-Hop Reasoning? (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) latently perform multi-hop reasoning with complex prompts such as "The mother of the singer of ‘Superstition’ is". |
| Approach: | They examine whether Large Language Models (LLMs) latently perform multi-hop reasoning with complex prompts such as "The mother of the singer of ‘Superstition’ is". |
| Outcome: | The proposed model can latently perform multi-hop reasoning with complex prompts such as "The mother of the singer of ‘Superstition’ is". |
Tracing and Dissecting How LLMs Recall Factual Knowledge for Real World Questions (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have shown promising ability to perform commonsense reasoning. |
| Approach: | They propose a two-dimensional analysis framework that incorporates token back-tracing and token decoding to uncover how LLMs conduct factual knowledge recall. |
| Outcome: | The proposed framework shows that LLMs lack relevant knowledge but struggle to select the most accurate information based on context during the retrieval and rerank phase. |
Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Increasing the ability of large language models to perform latent multihop reasoning is crucial for reducing the cost and deployment challenges. |
| Approach: | They propose an interpretability method that traces how logits propagate across layers and positions toward the final prediction. |
| Outcome: | The proposed method improves accuracy on five reasoning datasets. |
Reasoning or a Semblance of it? A Diagnostic Study of Transitive Reasoning in LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating Large Language Models (LLMs) on reasoning benchmarks demonstrates their ability to solve compositional questions. |
| Approach: | They investigate the transitive reasoning capabilities of two distinct LLM architectures, LLaMA 2 and Flan-T5, by manipulating facts within two compositional datasets: QASC and Bamboogle. |
| Outcome: | The proposed models leverage word/phrase overlaps across sections of test input, models’ inherent knowledge during pre-training or fine-tuning, and names of entities. |
How Long Reasoning Chains Influence LLMs’ Judgment of Answer Factuality (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly adopted as scalable judges for open-ended generation, yet how they form judgments remains insufficiently understood. |
| Approach: | They show that exposing reasoning influences LLM-based judgment . they also show that reasoning fluency and factuality critically shape judgment outcomes . |
| Outcome: | Empirical results show that the presence of reasoning significantly alters judgment behavior . stronger judges exhibit more selective behavior and achieve higher judgment accuracy . |
Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities in various tasks, but may rely on dataset biases as shortcuts for prediction. |
| Approach: | They propose to use a test suite to evaluate the impact of shortcuts on LLMs' performance. |
| Outcome: | The proposed test suite incorporates six shortcut types, five evaluation metrics, and four prompting strategies. |
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2025.naacl-long)
Copied to clipboard
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, Manaal Faruqui
| Challenge: | Recent advances in Large Language Models (LLMs) have significantly enhanced their capabilities across various cognitive tasks. |
| Approach: | They propose a high-quality evaluation dataset to test LLMs' ability to provide factual responses, assess retrieval capabilities, and evaluate the reasoning required to generate final answers. |
| Outcome: | The proposed framework improves performance in end-to-end RAG scenarios. |