| Challenge: | Existing numerical MRC models are weak in numerical reasoning, such as addition, subtraction, sorting and counting. |
| Approach: | They propose a numerical MRC model that integrates numerical reasoning into existing MRC models and achieves an EM-score of 64.56% on the DROP dataset. |
| Outcome: | The proposed model outperforms all existing machine reading comprehension models by considering the numerical relations among numbers on the DROP dataset. |
Similar Papers
Numerical reasoning in machine reading comprehension tasks: are we there yet? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Numerical reasoning based machine reading comprehension models have achieved near-human performance on a variety of benchmarks, but are they capable of learning to reason? |
| Approach: | They propose to use a DROP benchmark to measure machine reading comprehension and investigate models that have achieved near-human performance over standard metrics. |
| Outcome: | The DROP benchmark has inspired the design of specialized BERT and embedding the results into a specialized model. |
Question Directed Graph Attention Network for Numerical Reasoning over Text (2020.emnlp-main)
Copied to clipboard
Kunlong Chen, Weidi Xu, Xingyi Cheng, Zou Xiaochuan, Yuyu Zhang, Le Song, Taifeng Wang, Yuan Qi, Wei Chu
| Challenge: | Numerical reasoning requires both natural language understanding and arithmetic computation. |
| Approach: | They propose a graph representation for the context of the passage and question needed for numerical reasoning. |
| Outcome: | The proposed model achieves remarkable results in benchmark datasets such as DROP. |
Improving Numerical Reasoning Skills in the Modular Approach for Complex Question Answering on Text (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Neural Module Networks (NMNs) is an end-to-end differentiable model in the programmer-interpreter paradigm. |
| Approach: | They propose to make the interpreter question-aware and capture the relationship between entities and numbers in both questions and paragraphs. |
| Outcome: | The proposed models outperform the original models on the DROP dataset and are interpertable by nature. |
Teaching Neural Module Networks to Do Arithmetic (2022.coling-1)
Copied to clipboard
| Challenge: | Neural Module Networks (NMNs) have limited reasoning abilities and lack numerical reasoning capability. |
| Approach: | They propose to integrate the original question in the interpreter and introduce addition and subtraction modules that perform numerical reasoning over numbers. |
| Outcome: | The proposed methods outperform previous state-of-the-art models on a subset of DROP and achieve competitive reasoning performance. |
Have You Seen That Number? Investigating Extrapolation in Question Answering Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Numerical reasoning in machine reading comprehension (MRC) has shown drastic improvements over the past few years. |
| Approach: | They propose an E-digit number form that alleviates the lack of extrapolation in numerical MRC models. |
| Outcome: | The proposed model can't extrapolate to unseen numbers, the authors say . they also show that the model needs to treat numbers differently from regular words . |
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks (2022.acl-long)
Copied to clipboard
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, Ashwin Kalyan
| Challenge: | Existing AI systems fail to perform basic mathematical reasoning when presented in a slightly different manner. |
| Approach: | They propose a multi-task benchmark that evaluates the performance of AI systems on eight different tasks that at their core require simple arithmetic understanding. |
| Outcome: | The proposed benchmark compares the performance of AI systems on eight different tasks. |
Benchmarking Machine Reading Comprehension: A Psychological Perspective (2021.eacl-main)
Copied to clipboard
| Challenge: | MRC is a task that tests the ability of a machine to read and understand unstructured text. |
| Approach: | They propose a theoretical basis for the design of MRC datasets based on psychology and psychometrics and propose shortcut-proof questions and explanations as a part of the task design. |
| Outcome: | The proposed datasets should evaluate the model's ability to understand context-dependent situations and ensure substantive validity by shortcut-proof questions and explanation as a part of the task design. |
A Framework for Evaluation of Machine Reading Comprehension Gold Standards (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing literature on machine reading comprehension (MRC) data is limited on the data design of gold standards. |
| Approach: | They propose a framework to investigate linguistic features, lexical cues and ambiguity in MRC gold standards. |
| Outcome: | The proposed framework investigates the present linguistic features, required reasoning and background knowledge and factual correctness on the one hand, and the presence of lexical cues as a lower bound for the requirement of understanding on the other. |
Incorporating Syntax and Frame Semantics in Neural Network for Machine Reading Comprehension (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for machine reading comprehension rely on manually defined features and are difficult to generalize to other tasks. |
| Approach: | They propose a Syntax and Frame Semantics model for Machine Reading Comprehension which takes full advantage of syntax and frame semantics to get richer text representation. |
| Outcome: | The proposed model outperforms ten state-of-the-art models on machine reading comprehension tasks. |
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (N19-1)
Copied to clipboard
| Challenge: | a large body of work has highlighted the brittleness of reading comprehension systems . a crowdsourced, adversarially-created, 55k-question benchmark requires a more comprehensive understanding of paragraphs . |
| Approach: | They propose a reading comprehension benchmark that requires Discrete Reasoning over the content of paragraphs. |
| Outcome: | The proposed benchmarks show that the best systems only achieve 38.4% F1 on the generalized accuracy metric, while human performance is 96%. |