| Challenge: | Large Language Models (LLMs) latently perform multi-hop reasoning with complex prompts such as "The mother of the singer of ‘Superstition’ is". |
| Approach: | They examine whether Large Language Models (LLMs) latently perform multi-hop reasoning with complex prompts such as "The mother of the singer of ‘Superstition’ is". |
| Outcome: | The proposed model can latently perform multi-hop reasoning with complex prompts such as "The mother of the singer of ‘Superstition’ is". |
Similar Papers
Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts? (2025.findings-acl)
Copied to clipboard
| Challenge: | Latent multi-hop reasoning is a problem in Large Language Models that can develop shortcuts by encountering the head entity and answer entity in training sequences. |
| Approach: | They propose desiderata for shortcut-free evaluation of latent multi-hop reasoning ability . they exclude test queries where head and answer entities might have co-appeared . |
| Outcome: | The proposed model can latently recall and compose single-hop facts without shortcuts, but only for certain types of queries. |
Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers? (2024.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art Large Language Models (LLMs) are accredited with a number of different capabilities, including reading comprehension, mathematical and reasoning skills, and possessing scientific knowledge. |
| Approach: | They propose a benchmark to generate seemingly plausible multi-hop reasoning chains that ultimately lead to incorrect answers. |
| Outcome: | The proposed model circumvents the reasoning requirement but in subtle ways . it shows that it is more difficult to generate plausible alternatives . |
Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can solve complex multi-step problems, but little is known about how these computations are implemented internally. |
| Approach: | They propose a "back-patching" analysis method to solve multi-hop queries . they propose resolving the bridge entity into the bridge and the second hop into the target entity into latent steps. |
| Outcome: | The proposed method solves multi-hop queries that require two information extraction steps . it shows that the later layers lack the necessary knowledge to correctly generate the answer . |
Investigating Multi-Hop Factual Shortcuts in Knowledge Editing of Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent work has demonstrated the power of large language models in recalling knowledge and reasoning. |
| Approach: | They propose to erase shortcut neurons to mitigate the associated risks . 20% of the failures are attributed to shortcuts, they find . |
| Outcome: | The proposed approach reduces failures in multi-hop knowledge editing caused by shortcuts by 20% . |
Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Increasing the ability of large language models to perform latent multihop reasoning is crucial for reducing the cost and deployment challenges. |
| Approach: | They propose an interpretability method that traces how logits propagate across layers and positions toward the final prediction. |
| Outcome: | The proposed method improves accuracy on five reasoning datasets. |
Making Long-Context Language Models Better Multi-Hop Reasoners (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in long-context modeling have enhanced language models for complex tasks, but they struggle with multi-hop reasoning and noisy contexts. |
| Approach: | They propose an approach that prompts LMs to supply attributions for each assertion during reasoning. |
| Outcome: | The proposed model achieves competitive performance on multi-hop reasoning benchmarks, closely paralleling proprietary LMs such as ChatGPT and Claude-instant. |
There’s No Such Thing as Simple Reasoning for LLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing work has focused on relatively complex “many-hop” reasoning problems. |
| Approach: | They analyse the performance of fine-tuned LLMs on simple reasoning problems . they find the models remain highly brittle, being susceptible to seemingly innocent perturbations . |
| Outcome: | The proposed models fail on simple reasoning problems, but are highly brittle . they are susceptible to seemingly innocent perturbations, such as adding duplicates to the set of premises and shuffling the order in which the premises are presented. |
Conditional and Modal Reasoning in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Inferences involving conditionals and modals play a central role in the fundamental human ability to reason about distal possibilities. |
| Approach: | They propose to assess the extent to which 29 LLMs are able to distinguish logically correct inferences from logical fallacious ones. |
| Outcome: | The LLMs that make basic mistakes with conditionals and modals display inconsistent judgments across inference patterns involving epistemic modal and conditionals, and give answers about complex conditional inferences that do not match reported human judgments. |
Are Large Language Model Temporally Grounded? (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent large language models lack a consistent temporal model of textual narratives . sentence ordering in unlabelled texts is only weakly correlated with event ordering . |
| Approach: | They evaluate LLMs with textual narratives and evaluate their common-sense knowledge . they find that LLM models struggle the most with self-consistency . |
| Outcome: | The proposed models lack a consistent temporal model of textual narratives. |
A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners (2024.emnlp-main)
Copied to clipboard
Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie Su, Camillo Taylor, Dan Roth
| Challenge: | a new hypothesis-testing framework is developed to assess whether large language models possess genuine reasoning abilities or primarily depend on token bias. |
| Approach: | They propose a framework to assess whether large language models have genuine reasoning abilities or primarily depend on token bias. |
| Outcome: | The proposed framework outlines a list of hypotheses where token biases are readily identifiable . the results suggest that most LLMs still struggle with logical reasoning . |