ToW: Thoughts of Words Improve Reasoning in Large Language Models (2025.naacl-long)
Copied to clipboard
Zhikun Xu, Ming Shen, Jacob Dineen, Zhaonan Li, Xiao Ye, Shijie Lu, Aswin Rrv, Chitta Baral, Ben Zhou
| Challenge: | Unlike other data augmentation methods, thoughts of words (TOW) views next-word prediction as a core reasoning task and injects fine-grained thoughts into pre-training texts. |
| Approach: | They propose a training-time data-augmentation method called thoughts of words (TOW) that injects fine-grained thoughts directly into a next-word prediction task and teaches the model to understand how the observed next word is related to previous contexts. |
| Outcome: | The proposed method reduces model hallucination by 10% and improves reasoning performance by 7% to 9% on average. |
Similar Papers
Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Chain-of-thought (CoT) prompting is a technique to enhance the reasoning abilities of Large language models (LLMs) however, the reasoning chains of demonstrations are observed to be prone to errors, which can lead to incorrect reasoning during inference. |
| Approach: | They propose an iterative bootstrapping technique to enhance the reasoning abilities of Large language models (LLMs) by generating a series of reasoning steps to obtain the answer, and using the reasoning chains as exemplars to demonstrate the task. |
| Outcome: | The proposed method improves the performance of Large language models (LLMs) on three reasoning tasks on ten datasets. |
Let Retrievers Think Before Action: Thought-Augmented Embedding for Dense Retrieval (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models have demonstrated that explicit step-by-step thinking can substantially improve performance on complex tasks. |
| Approach: | They propose a model that generates preliminary thoughts for input queries before document retrieval. |
| Outcome: | The proposed model generates preliminary thoughts for input queries before document retrieval. |
Let’s Be Self-generated via Step by Step: A Curriculum Learning Approach to Automated Reasoning with Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing efforts to improve CoT prompting have limitations that require extensive human effort or performance needs to be improved. |
| Approach: | They propose a prompt approach for automatic reasoning called LBS3 inspired by curriculum learning which better reflects human learning habits. |
| Outcome: | The proposed approach achieves strongly competitive performance compared to baselines in reasoning-intensive tasks with varying open- and closed-source LLMs. |
CoT-RAG: Integrating Chain of Thought and Retrieval-Augmented Generation to Enhance Reasoning in Large Language Models (2025.findings-emnlp)
Copied to clipboard
Feiyang Li, Peng Fang, Zhan Shi, Arijit Khan, Fang Wang, Weihao Wang, null Zhangxin-hw, Cui Yongjian
| Challenge: | Chain-of-thought reasoning has two key limitations: lack of reliability when solely relying on LLM-generated reasoning chains and interference from natural language reasoning steps with the models’ inference logic. |
| Approach: | They propose a chain-of-thought reasoning framework with three key designs to address these issues. |
| Outcome: | The proposed framework improves the performance of large language models on complex tasks by incorporating knowledge graphs and learnable knowledge case-aware RAG. |
GoT: Effective Graph-of-Thought Reasoning in Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have been advancing at an unprecedented pace. |
| Approach: | They propose a graph-based approach which models human thought processes as a chain and as 'graphs' by representing thought units as nodes and connections between them as edges, they capture the non-sequential nature of human thinking and allows for a more realistic modeling of thought processes. |
| Outcome: | The proposed model improves on a text-only reasoning task and a multimodal reasoning task. |
How to represent a word and predict it, too: Improving tied architectures for language modelling (D18-1)
Copied to clipboard
| Challenge: | Recent state-of-the-art models use word embeddings as input and output mappings instead of tied models. |
| Approach: | They propose to decouple hidden state from word embedding prediction . they extend their proposed modification to word2vec models . |
| Outcome: | The proposed architectures achieve comparable or better results compared to previous models without tying . the proposed architecture reduces parameters, enabling more compact models and faster learning. |
PretrainRL: Alleviating Factuality Hallucination of Large Language Models at the Beginning (2026.findings-acl)
Copied to clipboard
Langming Liu, Kangtao Lv, Haibin Chen, Weidong Zhang, Yejing Wang, Shilei Liu, Xin Tong, Yujin Yuan, Yongwei Wang, Wenbo Su, Bo Zheng
| Challenge: | Large language models suffer from factual hallucinations where they generate verifiable falsehoods. |
| Approach: | They propose a framework that integrates reinforcement learning into the pretraining phase to consolidate factual knowledge. |
| Outcome: | The proposed framework significantly alleviates factual hallucinations and outperforms state-of-the-art methods. |
Thought calibration: Efficient and confident test-time scaling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for teaching language models to be economical with their token budgets have failed to achieve the desired results. |
| Approach: | They propose to calibrate a language model's growing body of thoughts to determine when new reasoning plateaus. |
| Outcome: | The proposed framework preserves model performance with up to 60% reduction in thinking tokens on in-distribution data, and up to 20% in out-of-difference data. |
Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs (2024.findings-acl)
Copied to clipboard
Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruirui Li, Xianfeng Tang, Suhang Wang, Yu Meng, Jiawei Han
| Challenge: | Existing studies suggest augmenting LLMs with external text corpora to alleviate hallucination problems. |
| Approach: | They propose to augment large language models with text units retrieved from external knowledge corpora to alleviate the issue. |
| Outcome: | The proposed framework outperforms baselines on GRBench with three LLMs and shows that iterative reasoning outperformed the baselines. |
A Survey of Inductive Reasoning for Large Language Models (2026.acl-long)
Copied to clipboard
Kedi Chen, Dezhao Ruan, Yuhao Dan, Yaoting Wang, Siyu Yan, Xuecheng Wu, Yinqi Zhang, Qin Chen, Jie Zhou, Liang He, Biqing Qi, Linyang Li, Qipeng Guo, Xiaoming Shi, Wei Zhang
| Challenge: | Inductive reasoning is an important task for large language models (LLMs). |
| Approach: | They propose a survey of inductive reasoning for large language models . they categorize methods into three main areas: post-training enhancement, test-time exploration, and data augmentation. |
| Outcome: | The proposed method improves inductive reasoning in large language models. |