| Challenge: | a key component of reading a passage is the ability to apply knowledge gained from the passage to a new situation. |
| Approach: | They propose a benchmark for reading comprehension targeting Reasoning Over Paragraph Effects in Situations. |
| Outcome: | The proposed model performs slightly better than randomly guessing an answer of the correct type, but is below the human performance of 89.0%. |
Similar Papers
Towards Interpretable Reasoning over Paragraph Effects in Situation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models ignore complex reasoning process and solve it with a one-step "black box" approach. |
| Approach: | They propose a sequential approach which explicitly models each step of the reasoning process with neural network modules. |
| Outcome: | The proposed model is more interpretable and more accurate than existing models. |
Reading Comprehension with Graph-based Temporal-Casual Reasoning (C18-1)
Copied to clipboard
| Challenge: | Existing methods for reading comprehension tasks ignore semantic relations between sentences or use sliding window scanning over the words of the passage without sentence breaks. |
| Approach: | They propose a method to integrate information from multiple sentences to answer complex questions. |
| Outcome: | Experiments on RACE and MCTest show that the proposed approach improves state-of-the-art methods on simple factoid questions. |
Multi-Step Inference for Reasoning Over Paragraphs (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for complex reasoning use symbols or black-box transformers . a compositional model can chain together free-form predicates and logical connectives . |
| Approach: | They propose a compositional model that finds relevant sentences and then chains them together using neural modules. |
| Outcome: | The proposed model improves performance on a recently-introduced dataset. |
Learning with Instance Bundles for Reading Comprehension (2021.emnlp-main)
Copied to clipboard
| Challenge: | a study shows that training reading comprehension models assumes that the training instances are independent and identically distributed . however, this assumption can cause the learner to ignore distinguishing cues between related or minimally different questions . |
| Approach: | They propose to normalize question-answer scores across neighborhoods of closely contrasting questions and/or answers by adding a cross entropy loss term to the supervision signal. |
| Outcome: | The proposed methods show up to 9% absolute gains in accuracy on two datasets. |
CRAB: Assessing the Strength of Causal Relationships Between Real-world Events (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for reasoning about events in narratives do not understand the complexity of the causal relationships of events in the narrative. |
| Approach: | They propose a Causal Reasoning Assessment Benchmark to evaluate causal understanding of events in narratives. |
| Outcome: | The proposed model performs worse when models are derived from complex causal structures than simple linear causal chains. |
Building Context-aware Clause Representations for Situation Entity Type Classification (D18-1)
Copied to clipboard
| Challenge: | Existing models for categorizing clauses based on situation entity types do not provide accurate results. |
| Approach: | They propose to build context-aware clause representations for predicting situation entity types of clauses by modeling context influences and inter-dependencies of clause. |
| Outcome: | The proposed model achieves state-of-the-art performance on genre-rich dataset MASC+Wiki . |
Commonsense Reasoning for Natural Language Processing (2020.acl-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we will outline the various types of commonsense knowledge and discuss techniques to gather and represent commonsence knowledge. |
| Approach: | This tutorial will provide researchers with the critical foundations and recent advances in commonsense representation and reasoning. |
| Outcome: | This tutorial will outline the various types of commonsense and discuss techniques to gather and represent commonsence knowledge while highlighting the challenges specific to this type of knowledge (e.g., reporting bias). |
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (N19-1)
Copied to clipboard
| Challenge: | a large body of work has highlighted the brittleness of reading comprehension systems . a crowdsourced, adversarially-created, 55k-question benchmark requires a more comprehensive understanding of paragraphs . |
| Approach: | They propose a reading comprehension benchmark that requires Discrete Reasoning over the content of paragraphs. |
| Outcome: | The proposed benchmarks show that the best systems only achieve 38.4% F1 on the generalized accuracy metric, while human performance is 96%. |
The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can explain quantum mechanics and write sophisticated code, yet fail to parse sentences like "The cat that the mouse feared chased meowed" |
| Approach: | They propose a framework to distinguish structural understanding from semantic pattern matching . they use a set of 9,720 comprehension questions on center-embedded sentences . |
| Outcome: | a new framework shows that models lose performance when they abandon structural analysis for semantic associations. |
What-if I ask you to explain: Explaining the effects of perturbations in procedural text (2020.findings-emnlp)
Copied to clipboard
| Challenge: | QUARTET constructs explanations from paragraphs using procedural text . qartet achieves 18 points better on explanation accuracy compared to strong baselines on a recent process comprehension benchmark. |
| Approach: | They propose a system that constructs explanations from paragraphs by modeling the explanation task as a multitask learning problem. |
| Outcome: | The proposed system achieves 18 points better on explanation accuracy compared to strong baselines on a process comprehension benchmark. |