Papers by Julius Steen
Abstractive Timeline Summarization (D19-54)
Copied to clipboard
| Challenge: | Prior approaches to TLS focus on extractive methods, which generate extractive timelines . a study with human judges shows that our abstractive system also produces output that is easy to read and understand. |
| Approach: | They propose an abstractive timeline summarization system that is unsupervised . their system outperforms extractive systems in terms of ROUGE scores . |
| Outcome: | The proposed system outperforms extractive systems in terms of ROUGE scores . it produces output that is easy to read and understand, the authors say . |
Bias in News Summarization: Measures, Pitfalls and Corpora (2024.findings-acl)
Copied to clipboard
| Challenge: | Pretrained large language models can reproduce harmful social biases in constrained settings, such as summarization. |
| Approach: | They propose a method to generate input documents with carefully controlled demographic attributes and then apply it to a controlled setting. |
| Outcome: | The proposed method allows to generate input documents with carefully controlled demographic attributes while working with real-world input documents. |
How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluation (2021.eacl-main)
Copied to clipboard
| Challenge: | Current manual evaluation methods for text summarization have low correlation with human judgements on summary quality. |
| Approach: | They conduct two evaluation experiments on two aspects of summaries’ linguistic quality (coherence and repetitiveness) they find that study parameters such as the total number of annotators and distribution of annotes to annotation items are often not fully reported. |
| Outcome: | The proposed methods can inflate type I errors up to eight-fold and the overall number of annotators can have a strong impact on study power. |
With a Little Push, NLI Models can Robustly and Efficiently Predict Faithfulness (2023.acl-short)
Copied to clipboard
| Challenge: | Conditional language models generate unfaithful output that is not supported by their input . this jeopardizes trust in real-world applications, raising a need for automatic faithfulness metrics. |
| Approach: | They propose to augment conditional language models with robust inference procedures to improve faithfulness. |
| Outcome: | The proposed approach outperforms existing models on the TRUE benchmark. |
How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to evaluate summary coherence are often evaluated using disparate datasets and metrics. |
| Approach: | They propose to use automatic evaluation to evaluate coherence of summaries by selecting high-scoring candidates. |
| Outcome: | The proposed methods show that they can perform better on an even playing field. |