Ping Yu, Tianlu Wang, Olga Golovneva, Badr AlKhamissi, Siddharth Verma, Zhijing Jin, Gargi Ghosh, Mona Diab, Asli Celikyilmaz
| Challenge: | Large language models have shown increasing in-context learning capabilities with scaling up the model and data sizes. |
| Approach: | They propose a benchmark and suite of analyses to evaluate reasoning skills of large language models. |
| Outcome: | The proposed model compares pre-trained and fine-tuned models on tasks that require reasoning skills to solve. |
Similar Papers
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2024.naacl-long)
Copied to clipboard
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, Yoon Kim
| Challenge: | Recent language models possess impressive performance across a wide range of tasks . however, they often rely on narrow, non-transferable procedures for task-solving . |
| Approach: | They propose to evaluate language models using "counterfactual" task variants that deviate from standard tasks. |
| Outcome: | The proposed framework shows that language models perform better on a wide range of tasks compared to the default conditions. |
oLMpics-On What Language Model Pre-training Captures (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent success of pre-trained language models has spurred widespread interest in their capabilities. |
| Approach: | They propose an evaluation protocol that includes zero-shot evaluation and no fine-tuning . they propose to compare the learning curve of a fine- tuned LM to the learning of multiple controls . |
| Outcome: | The proposed evaluation protocol compares the learning curve of a fine-tuned LM to the learning of multiple controls. |
Complex Reasoning in Natural Language (2023.acl-tutorials)
Copied to clipboard
| Challenge: | Recent research shows that pretrained language models are often brittle for complex reasoning tasks. |
| Approach: | They propose to use pre-trained language models to teach machines to reason over texts . they will review recent promising approaches to tackling complex reasoning tasks . |
| Outcome: | This tutorial reviews promising approaches to complex reasoning tasks . it reviews the methods that can be used to augment models with robustness . |
Do Language Embeddings capture Scales? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained Language Models possess significant linguistic, common sense and factual knowledge, but are short of the capability required for general common-sense reasoning. |
| Approach: | They propose to train pretrained language models with a method of canonicalizing numbers . they address a task which is also pre-requisite for general common-sense reasoning . |
| Outcome: | The proposed model can answer questions about common sense and linguistics, but lacks the capability to answer questions on scalar attributes. |
Infusing Finetuning with Semantic Dependencies (2021.tacl-1)
Copied to clipboard
| Challenge: | Several diagnostics help to localize the benefits of our approach. |
| Approach: | They apply convolutional graph encoders to integrate semantic parses into task-specific finetuning. |
| Outcome: | The proposed approach yields benefits to natural language understanding (NLU) tasks in the GLUE benchmark. |
Training in Step-by-Step Formal Reasoning Improves Pronominal Reasoning in Language Models (2026.eacl-short)
Copied to clipboard
| Challenge: | Large reasoning models are limited to formal reasoning, i.e., math, code, and logic. |
| Approach: | They evaluate a set of large reasoning models on a dataset for pronoun resolution and fidelity. |
| Outcome: | The results show that distilling step-by-step formal reasoning improves pronoun resolution and fidelity. |
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020.acl-main)
Copied to clipboard
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, Samuel R. Bowman
| Challenge: | Unsupervised pretraining has recently pushed the state of the art on many natural language understanding tasks. |
| Approach: | They perform a large-scale survey on a pretrained RoBERTa model with 110 intermediate-target task combinations and 25 probing tasks to reveal the specific skills that drive transfer. |
| Outcome: | The proposed model is trained on 110 intermediate-target task combinations and compared with 25 probing tasks to reveal the specific skills that drive transfer. |
Out-of-Context Reasoning in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lightweight technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
| Approach: | They propose a lightweight technique that trains only new token embeddings on axioms . they train only new embeddables and evaluate them on unseen tasks . |
| Outcome: | The proposed technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
Enhancing Large Language Models through Transforming Reasoning Problems into Classification Tasks (2024.lrec-main)
Copied to clipboard
Tarun Raheja, Raunak Sinha, Advit Deepak, Will Healy, Jayanth Srinivasa, Myungjin Lee, Ramana Kompella
| Challenge: | Existing approaches to improve LLMs' reasoning capabilities for constraint satisfaction problems (CSPs) are needed to solve complex tasks. |
| Approach: | They propose a method that leverages the LLM's ability to decide when to call a function from a set of logical-linguistic primitives, each of which can interact with a local “scratchpad” memory and logical inference engine. |
| Outcome: | The proposed method improves the reasoning capabilities of large language models for constraint satisfaction problems by 40% over baselines. |
When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracy (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent Large Reasoning Models (LRMs) with thinking traces have shown strong performance on English reasoning tasks. |
| Approach: | They evaluate two leading LRMs with thinking traces on established benchmark XReasoning and propose directions for future research. |
| Outcome: | The proposed models often revert to English or produce fragmented reasoning in other languages, revealing a substantial gap in the capability of thinking in non-English languages. |