Numerical reasoning in machine reading comprehension tasks: are we there yet? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Numerical reasoning based machine reading comprehension models have achieved near-human performance on a variety of benchmarks, but are they capable of learning to reason? |
| Approach: | They propose to use a DROP benchmark to measure machine reading comprehension and investigate models that have achieved near-human performance over standard metrics. |
| Outcome: | The DROP benchmark has inspired the design of specialized BERT and embedding the results into a specialized model. |
Similar Papers
NumNet: Machine Reading Comprehension with Numerical Reasoning (D19-1)
Copied to clipboard
| Challenge: | Existing numerical MRC models are weak in numerical reasoning, such as addition, subtraction, sorting and counting. |
| Approach: | They propose a numerical MRC model that integrates numerical reasoning into existing MRC models and achieves an EM-score of 64.56% on the DROP dataset. |
| Outcome: | The proposed model outperforms all existing machine reading comprehension models by considering the numerical relations among numbers on the DROP dataset. |
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (N19-1)
Copied to clipboard
| Challenge: | a large body of work has highlighted the brittleness of reading comprehension systems . a crowdsourced, adversarially-created, 55k-question benchmark requires a more comprehensive understanding of paragraphs . |
| Approach: | They propose a reading comprehension benchmark that requires Discrete Reasoning over the content of paragraphs. |
| Outcome: | The proposed benchmarks show that the best systems only achieve 38.4% F1 on the generalized accuracy metric, while human performance is 96%. |
Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension (D19-1)
Copied to clipboard
| Challenge: | End-to-end reading comprehension models have been successful at extracting text answers, but there are still problems with generalizing them to abstractive numerical reasoning. |
| Approach: | They propose to augment a BERT-based reading comprehension model with a set of executable ‘programs’ which encompass simple arithmetic as well as extraction. |
| Outcome: | The proposed model can perform 33% absolute improvement on the DROP dataset, with very few training examples. |
Benchmarking Machine Reading Comprehension: A Psychological Perspective (2021.eacl-main)
Copied to clipboard
| Challenge: | MRC is a task that tests the ability of a machine to read and understand unstructured text. |
| Approach: | They propose a theoretical basis for the design of MRC datasets based on psychology and psychometrics and propose shortcut-proof questions and explanations as a part of the task design. |
| Outcome: | The proposed datasets should evaluate the model's ability to understand context-dependent situations and ensure substantive validity by shortcut-proof questions and explanation as a part of the task design. |
How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks (D18-1)
Copied to clipboard
| Challenge: | Recent research addresses reading comprehension, where examples consist of (question, passage, answer) tuples. |
| Approach: | They establish sensible baselines for bAbI, SQuAD, CBT, CNN and Who-did-What datasets and compare them to their previous work. |
| Outcome: | The proposed models perform on 14 out of 20 bAbI, SQuAD, CBT, CNN and Who-did-What datasets. |
Have You Seen That Number? Investigating Extrapolation in Question Answering Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Numerical reasoning in machine reading comprehension (MRC) has shown drastic improvements over the past few years. |
| Approach: | They propose an E-digit number form that alleviates the lack of extrapolation in numerical MRC models. |
| Outcome: | The proposed model can't extrapolate to unseen numbers, the authors say . they also show that the model needs to treat numbers differently from regular words . |
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks (2022.acl-long)
Copied to clipboard
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, Ashwin Kalyan
| Challenge: | Existing AI systems fail to perform basic mathematical reasoning when presented in a slightly different manner. |
| Approach: | They propose a multi-task benchmark that evaluates the performance of AI systems on eight different tasks that at their core require simple arithmetic understanding. |
| Outcome: | The proposed benchmark compares the performance of AI systems on eight different tasks. |
On Making Reading Comprehension More Comprehensive (D19-58)
Copied to clipboard
| Challenge: | Getting machines to "understand" text is a vast and long-standing problem, made more challenging by the fact that it is not even clear what it means to understand text. |
| Approach: | They propose a question-based approach to machine reading comprehension that uses a natural language question to test a system's comprehension of a passage of text. |
| Outcome: | The proposed questions have surface cues or other biases that allow a model to shortcut the intended reasoning process. |
Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent benchmarks have assessed language models' numerical abilities . limitations include tokenization and representation of numbers in text, hallucination, and a lack of numerical commonsense knowledge. |
| Approach: | They propose a hierarchical taxonomy for numerical reasoning skills that includes representation, number sense, manipulation, and complex reasoning. |
| Outcome: | The proposed model outperforms other models on the tabular Natural Language Inference task. |
Question Directed Graph Attention Network for Numerical Reasoning over Text (2020.emnlp-main)
Copied to clipboard
Kunlong Chen, Weidi Xu, Xingyi Cheng, Zou Xiaochuan, Yuyu Zhang, Le Song, Taifeng Wang, Yuan Qi, Wei Chu
| Challenge: | Numerical reasoning requires both natural language understanding and arithmetic computation. |
| Approach: | They propose a graph representation for the context of the passage and question needed for numerical reasoning. |
| Outcome: | The proposed model achieves remarkable results in benchmark datasets such as DROP. |