| Challenge: | Existing models for numerical reasoning are limited by their flexibility and require specialized architectures to capture high-level skills. |
| Approach: | They propose to inject numerical reasoning skills into pre-trained LMs by generating large amounts of data and training in a multi-task setup. |
| Outcome: | The proposed model performs better on DROP than other models of comparable size while maintaining high performance on standard RC tasks. |
Similar Papers
Do Language Embeddings capture Scales? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained Language Models possess significant linguistic, common sense and factual knowledge, but are short of the capability required for general common-sense reasoning. |
| Approach: | They propose to train pretrained language models with a method of canonicalizing numbers . they address a task which is also pre-requisite for general common-sense reasoning . |
| Outcome: | The proposed model can answer questions about common sense and linguistics, but lacks the capability to answer questions on scalar attributes. |
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained language models are currently used for many NLP tasks . however, inference for these models requires significant computational resources . |
| Approach: | They propose to use a shared text encoder to amortize the computational cost of inference over multiple tasks. |
| Outcome: | The proposed method reduces the size of the extracted representations by a factor of 16 to store them for later use. |
Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2022.acl-long)
Copied to clipboard
| Challenge: | Large pre-trained language models struggle in tasks that require reasoning . recent work shows that they struggle in performing symbolic reasoning operations without substantial amounts of additional data. |
| Approach: | They propose to leverage semi-structured tables and generate at scale question-paragraph pairs where answering the question requires reasoning over multiple facts in the paragraph. |
| Outcome: | The proposed model outperforms T5, a popular pre-trained encoder-decoder model, on reasoning-focused reading comprehension datasets. |
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference (2023.acl-short)
Copied to clipboard
| Challenge: | Large-scale language model pretraining is expensive as the models and pretraining corpora have become larger over time. |
| Approach: | They propose a modified transformer encoder that increases throughput for masked language model pretraining by more than 2x. |
| Outcome: | The proposed model increases throughput on IMDB and Amazon reviews classification and CoNLL NER tasks by 3.5x with minimal performance degradation. |
Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks (2026.acl-short)
Copied to clipboard
| Challenge: | Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. |
| Approach: | They propose a framework that integrates Language Learning Tasks alongside standard next-token prediction to stimulate the acquisition of morphological, syntactic, and semantic knowledge. |
| Outcome: | The proposed framework improves performance on linguistic competence benchmarks while maintaining competitive performance on reasoning tasks. |
Learning Multi-Step Reasoning by Solving Arithmetic Tasks (2023.acl-short)
Copied to clipboard
| Challenge: | Recent studies have demonstrated large LMs’ impressive performance in solving math problems, but such ability seems only to emerge from models with abundant parameters. |
| Approach: | They propose to continuously pre-train LMs with the capabilities of multi-step reasoning by continuously pretraining them on a synthetic dataset MsAT. |
| Outcome: | The proposed method improves LMs' multi-step reasoning abilities on four math word problem datasets. |
Scaling Synthetic Logical Reasoning Datasets with Context-Sensitive Declarative Grammars (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing proof generation algorithms bias reasoning toward specific proof traces and limit extensibility. |
| Approach: | They propose a framework with flexible context-sensitive rules binding multiple languages . they propose to use English verbalization of predicates to enhance logical reasoning . |
| Outcome: | The proposed framework surpasses GPT-4 in accuracy on a human-authored logic dataset by 12%. |
Enhancing Large Language Models through Transforming Reasoning Problems into Classification Tasks (2024.lrec-main)
Copied to clipboard
Tarun Raheja, Raunak Sinha, Advit Deepak, Will Healy, Jayanth Srinivasa, Myungjin Lee, Ramana Kompella
| Challenge: | Existing approaches to improve LLMs' reasoning capabilities for constraint satisfaction problems (CSPs) are needed to solve complex tasks. |
| Approach: | They propose a method that leverages the LLM's ability to decide when to call a function from a set of logical-linguistic primitives, each of which can interact with a local “scratchpad” memory and logical inference engine. |
| Outcome: | The proposed method improves the reasoning capabilities of large language models for constraint satisfaction problems by 40% over baselines. |
Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent benchmarks have assessed language models' numerical abilities . limitations include tokenization and representation of numbers in text, hallucination, and a lack of numerical commonsense knowledge. |
| Approach: | They propose a hierarchical taxonomy for numerical reasoning skills that includes representation, number sense, manipulation, and complex reasoning. |
| Outcome: | The proposed model outperforms other models on the tabular Natural Language Inference task. |
Pre-trained language model representations for language generation (N19-1)
Copied to clipboard
| Challenge: | Pre-trained language model representations have been successful in a wide range of language understanding tasks. |
| Approach: | They propose to use pre-trained language model representations to integrate them into sequence to sequence models and apply it to machine translation and abstractive summarization. |
| Outcome: | The proposed model is able to perform 5.3 BLEU in machine translation and 5.3 on the full text version of CNN/DailyMail. |