| Challenge: | Summarization is a well-established method of measuring reading proficiency in traditional English as a second or other language assessments. |
| Approach: | They propose three approaches to automatically assess learner summary for evaluating non-native reading comprehension using a summarization task and a long-term memory model. |
| Outcome: | The proposed models outperform traditional methods and produce quality assessments close to professional examiners. |
Similar Papers
On Making Reading Comprehension More Comprehensive (D19-58)
Copied to clipboard
| Challenge: | Getting machines to "understand" text is a vast and long-standing problem, made more challenging by the fact that it is not even clear what it means to understand text. |
| Approach: | They propose a question-based approach to machine reading comprehension that uses a natural language question to test a system's comprehension of a passage of text. |
| Outcome: | The proposed questions have surface cues or other biases that allow a model to shortcut the intended reasoning process. |
SummEval: Re-evaluating Summarization Evaluation (2021.tacl-1)
Copied to clipboard
Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, Dragomir Radev
| Challenge: | a lack of comprehensive studies on evaluation metrics for text summarization hinders progress . a new study aims to improve evaluation metrics that correlate with human judgments . |
| Approach: | They propose to re-evaluate automatic evaluation metrics and share a toolkit for evaluation . they hope to promote a more complete evaluation protocol for text summarization . |
| Outcome: | The proposed evaluation metrics are inconsistent with existing evaluation protocols. |
How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to evaluate summary coherence are often evaluated using disparate datasets and metrics. |
| Approach: | They propose to use automatic evaluation to evaluate coherence of summaries by selecting high-scoring candidates. |
| Outcome: | The proposed methods show that they can perform better on an even playing field. |
How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks (D18-1)
Copied to clipboard
| Challenge: | Recent research addresses reading comprehension, where examples consist of (question, passage, answer) tuples. |
| Approach: | They establish sensible baselines for bAbI, SQuAD, CBT, CNN and Who-did-What datasets and compare them to their previous work. |
| Outcome: | The proposed models perform on 14 out of 20 bAbI, SQuAD, CBT, CNN and Who-did-What datasets. |
Summary Explorer: Visualizing the State of the Art in Text Summarization (2021.emnlp-demo)
Copied to clipboard
| Challenge: | Automatic text summarization is the task of generating a summary of a long text by condensing it to its most important parts. |
| Approach: | They propose a tool to visually explore document summarization systems based on three well-known summary quality criteria . |
| Outcome: | The proposed tool compiles outputs of 55 state-of-the-art document summarization approaches and visually explores them during a qualitative assessment. |
Comprehensive Multi-Dataset Evaluation of Reading Comprehension (D19-58)
Copied to clipboard
| Challenge: | Recent research aims to facilitate training and evaluation on several reading comprehension datasets at the same time. |
| Approach: | They propose an evaluation server that reports performance on seven diverse reading comprehension datasets and includes synthetic augmentations to test models' ability to handle out-of-domain questions. |
| Outcome: | The evaluation server performs on seven reading comprehension datasets, and collects and includes synthetic augmentations for these datasets to test models' ability to handle out-of-domain questions. |
Summarization Evaluation in the Absence of Human Model Summaries Using the Compositionality of Word Embeddings (C18-1)
Copied to clipboard
| Challenge: | Existing summary evaluation methods rely on multiple model summaries to evaluate quality of summary outputs. |
| Approach: | They propose a new summary evaluation approach that does not require human model summaries . they exploit compositional capabilities of word embeddings to develop features . |
| Outcome: | The proposed metric replicates human-generated summarization scores on data from TAC 2008 and 2009 . the features are then used to train a learning model for predicting the summary content quality in the absence of gold models. |
Neural Text Summarization: A Critical Evaluation (D19-1)
Copied to clipboard
| Challenge: | Current approaches to text summarization use advanced attention and copying mechanisms, multi-task and multi-reward training techniques. |
| Approach: | They evaluate datasets, evaluation metrics, and models for text summarization . they highlight three primary shortcomings: 1) datasets leave task underconstrained; 2) models overfit layout biases . |
| Outcome: | The current evaluation protocol is weakly correlated with human judgment and does not account for factual correctness. |
Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation (2023.emnlp-main)
Copied to clipboard
Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev
| Challenge: | Compared to neural systems, automatic metrics should be interpretable and provide intuitive insights into system performance and output quality. |
| Approach: | They propose to use a two-stage evaluation pipeline to extract basic information units from one text sequence and check the extracted units in another sequence. |
| Outcome: | The proposed metrics can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability. |
Searching for Effective Neural Extractive Summarization: What Works and What’s Next (P19-1)
Copied to clipboard
| Challenge: | Recent years have seen success in the use of deep neural networks on text summarization, but there is no clear understanding of why they perform so well or how they might be improved. |
| Approach: | They propose to use different types of model architectures to improve extractive summarization systems. |
| Outcome: | The proposed framework achieves state-of-the-art on CNN/DailyMail by a large margin based on observations and analysis. |