Automating Idea Unit Segmentation and Alignment for Assessing Reading Comprehension via Summary Protocol Analysis (2022.lrec-1)
Copied to clipboard
| Challenge: | In second language learning, summaries are among the most popular type of student assignments. |
| Approach: | They propose to revise the annotation guidelines to allow machine implementation of the new annotation guidelines. |
| Outcome: | The proposed algorithm achieves 0.789 precision and 0.844 recall over the L2WS 2021 corpus. |
Similar Papers
Align then Summarize: Automatic Alignment Methods for Summarization Corpus Creation (2020.lrec-1)
Copied to clipboard
| Challenge: | Summarizing text is not a straightforward task. |
| Approach: | They propose to use automated transcriptions to generate reports from automatic transcriptions as a dataset for neural summarization. |
| Outcome: | The proposed model improves on publicmeetings corpus on a dataset of aligned public meetings. |
Segmenting Natural Language Sentences via Lexical Unit Analysis (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on sequence segmentation models suffer from invalid predictions and a lack of consistency. |
| Approach: | They propose a unified span-based model that embeds every span and computes a score for each segmentation candidate. |
| Outcome: | The proposed model achieves state-of-the-art on 6 of the 3 tasks tested. |
On the Role of Summary Content Units in Text Summarization Evaluation (2024.naacl-short)
Copied to clipboard
Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo Walenta, Juri Opitz, Leonardo Ribeiro, João Sedoc, Daniel Deutsch, Simon Mille, Yixin Liu, Sebastian Gehrmann, Lining Zhang, Saad Mahamood, Miruna Clinciu, Khyathi Chandu, Yufang Hou
| Challenge: | a human written summary content unit (SCU) is used to judge the quality of a summary . a pyramid evaluation method is based on SCUs that decompose a reference summary into concise sentences . |
| Approach: | They propose to use automated SCUs to evaluate the quality of a candidate summary . they propose to generate SCU approximations from AMR meaning representations and large language models . |
| Outcome: | The proposed method can be fully automated, but lacks the human effort to validate it. |
Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation (2023.emnlp-main)
Copied to clipboard
Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev
| Challenge: | Compared to neural systems, automatic metrics should be interpretable and provide intuitive insights into system performance and output quality. |
| Approach: | They propose to use a two-stage evaluation pipeline to extract basic information units from one text sequence and check the extracted units in another sequence. |
| Outcome: | The proposed metrics can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability. |
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2023.acl-long)
Copied to clipboard
Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, Dragomir Radev
| Challenge: | Existing studies for summarization evaluation exhibit low inter-annotator agreement or lack scale. |
| Approach: | They propose a modified summarization salience protocol based on fine-grained semantic units and a robust summarizing evaluation benchmark. |
| Outcome: | The proposed protocol is based on fine-grained semantic units and allows for high inter-annotator agreement. |
Automatic Pyramid Evaluation Exploiting EDU-based Extractive Reference Summaries (D18-1)
Copied to clipboard
| Challenge: | Existing methods for evaluating content are not accurate because they only confirm if the summary contains small textual fragments. |
| Approach: | They propose to transform human-made reference summaries into extractive reference sums and weight them using elementary discourse units. |
| Outcome: | The proposed method strongly correlates with manual evaluations on DUC and TAC data sets. |
Beyond Agreement: Diagnosing the Rationale Alignment of Automated Essay Scoring Methods based on Linguistically-informed Counterfactuals (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing Automated Essay Scoring (AES) methods focus on sentence-level features, whereas Large Language Models (LLMs) are sensitive to conventions & accuracy, language complexity, and organization. |
| Approach: | They propose to use large language models to aid in decision-making . they propose to analyze the reasoning of neural models by analyzing sentence-level features. |
| Outcome: | The proposed method improves understanding of neural approaches to Automated Essay Scoring (AES) and can also apply to other domains seeking transparency in model-driven decisions. |
SummaCoz: A Dataset for Improving the Interpretability of Factual Consistency Detection for Summarization (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Summarization is an important application of Large Language Models. |
| Approach: | They integrate human-annotated and model-generated natural language explanations to elucidate how a summary deviates and becomes inconsistent with its source article. |
| Outcome: | The proposed model provides rationales for its judgments and improves its accuracy significantly. |
Automatic Evaluation of Language Generation Technology Based on Structure Alignment (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for automatic evaluation ignore syntax of sentences despite its importance in determining meaning. |
| Approach: | They propose an automatic evaluation metric that considers both the words in sentences and their syntactic structures. |
| Outcome: | The proposed method is comparable to baselines from two NLP tasks. |
Automatic Focus Annotation: Bringing Formal Pragmatics Alive in Analyzing the Information Structure of Authentic Data (N18-1)
Copied to clipboard
| Challenge: | Using focus-background dichotomy, discourse and information structure of sentences are being studied in context. |
| Approach: | They propose to automate the analysis of focus in authentic written data by using a range of lexical, syntactic, and semantic features to achieve an accuracy of 78.1%. |
| Outcome: | The proposed approach achieves 78.1% accuracy for identifying focus in authentic written data. |