| Challenge: | Existing pretraining languages such as ELMo and GPT have advanced a wide range of tasks. |
| Approach: | They propose a novel document-level encoder based on BERT which can express the semantics of a document and obtain representations for its sentences. |
| Outcome: | The proposed model achieves state-of-the-art in extractive and abstractive settings. |
Similar Papers
Abstractive Text Summarization based on Language Model Conditioning and Locality Modeling (2020.lrec-1)
Copied to clipboard
Dmitrii Aksenov, Julian Moreno-Schneider, Peter Bourgonje, Robert Schwarzenberg, Leonhard Hennig, Georg Rehm
| Challenge: | Abstractive summarization is an NLP task with many real-world applications. |
| Approach: | They propose to use a pre-trained language model to train a Transformer-based neural model . they propose a new method of BERT-windowing to allow chunk-wise processing of texts longer than the BERT window size . |
| Outcome: | The proposed model outperforms baseline models on CNN/Daily Mail dataset and shows its superiority on German dataset. |
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (N19-1)
Copied to clipboard
| Challenge: | Existing language representation models pre-train deep bidirectional representations from unlabeled text without significant task-specific architecture modifications. |
| Approach: | They propose a language representation model that pre-trains bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. |
| Outcome: | The proposed model achieves state-of-the-art results on eleven natural language processing tasks, pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement) |
Stepwise Extractive Summarization and Planning with Structured Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to extractive summarization use transformers to learn the structure of long inputs. |
| Approach: | They propose encoder-centric stepwise models for extractive summarization using structured transformers – HiBERT and Extended Transformers . |
| Outcome: | The proposed models outperform previous models on CNN/DailyMail extractive summarization and Rotowire table-to-text generation. |
Considering Nested Tree Structure in Sentence Extractive Summarization with Pre-trained Transformer (2021.emnlp-main)
Copied to clipboard
| Challenge: | Sentence extractive summarization shortens a document by selecting sentences for a summary while preserving its important contents. |
| Approach: | They propose a nested tree-based extractive summarization model on RoBERTa that uses syntactic and discourse trees to represent sentences in a given document. |
| Outcome: | The proposed model outperforms baseline models on the CNN/DailyMail dataset and achieves significantly better scores than the baseline models in terms of coherence and comparable scores to the state-of-the-art models. |
HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization (P19-1)
Copied to clipboard
| Challenge: | Neural extractive summarization models employ hierarchical encoders with inaccurate sentence-level labels. |
| Approach: | They propose a method to pre-train a hierarchical encoder with unlabeled data. |
| Outcome: | The proposed model outperforms its initialized counterpart by 1.25 ROUGE on CNN and 2.0 ROUGEE on a version of New York Times dataset. |
Document-Level Neural Machine Translation Using BERT as Context Encoder (2020.aacl-srw)
Copied to clipboard
| Challenge: | Large-scale pre-trained representations such as BERT have been widely used in many natural language understanding tasks. |
| Approach: | They propose to use BERT as a context encoder to achieve document-level contextual information, which is then integrated into both the encoder and decoder. |
| Outcome: | The proposed model outperforms strong document-level machine translation baselines on BLEU score and captures document- level context information to boost translation performance. |
On Extractive and Abstractive Neural Document Summarization with Transformer Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | We present a method to produce abstractive summaries of documents that exceed several thousand words . we compare transformer based methods to extractive methods, but extractive models score higher . |
| Approach: | They propose a method to generate abstractive summaries of documents that exceed several thousand words via neural abstractive summary. |
| Outcome: | The proposed method produces abstractive summaries of documents that exceed several thousand words . it is compared with baseline methods, state-of-the-art models and variants of the proposed method . |
Pretrained Transformers for Text Ranking: BERT and Beyond (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial provides an overview of text ranking using neural network architectures known as transformers. |
| Approach: | This tutorial provides an overview of text ranking with neural network architectures known as transformers. |
| Outcome: | This tutorial provides an overview of text ranking with neural network architectures known as transformers. |
Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for document summarization use graphs and unlabeled documents . Existing models require labeled data, and it is expensive to create summarized documents. |
| Approach: | They propose to rank sentences using transformer attentions and pre-training objectives by unlabeled documents. |
| Outcome: | The proposed model achieves state-of-the-art on unsupervised summarization and is less dependent on sentence positions. |
Summary Level Training of Sentence Rewriting for Abstractive Summarization (D19-54)
Copied to clipboard
| Challenge: | Existing models rely on sentence-level rewards or suboptimal labels to achieve summary-level ROUGE scores. |
| Approach: | They propose a model that extracts salient sentences from a document and paraphrases them to generate a summary. |
| Outcome: | The proposed model improves on CNN/Daily Mail and New York Times datasets. |