Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization (2023.acl-long)
Copied to clipboard
Pengcheng He, Baolin Peng, Song Wang, Yang Liu, Ruochen Xu, Hany Hassan, Yu Shi, Chenguang Zhu, Wayne Xiong, Michael Zeng, Jianfeng Gao, Xuedong Huang
| Challenge: | Z-Code++ is a pre-trained language model optimized for abstractive text summarization. |
| Approach: | They propose a pre-trained language model optimized for abstractive text summarization that uses a two-phase pre-training technique to improve model's performance. |
| Outcome: | The proposed model outperforms the competing models on low-resource summarization tasks in zero-shot and few-shot settings. |
Similar Papers
Pre-trained language model representations for language generation (N19-1)
Copied to clipboard
| Challenge: | Pre-trained language model representations have been successful in a wide range of language understanding tasks. |
| Approach: | They propose to use pre-trained language model representations to integrate them into sequence to sequence models and apply it to machine translation and abstractive summarization. |
| Outcome: | The proposed model is able to perform 5.3 BLEU in machine translation and 5.3 on the full text version of CNN/DailyMail. |
Novel Natural Language Summarization of Program Code via Leveraging Multiple Input Representations (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on code summarization shows that code descriptions are difficult to generate for developers unfamiliar with the code base. |
| Approach: | They propose a multi-task approach that trains two similar tasks to generate code descriptions for each line of code. |
| Outcome: | The proposed model improves over baselines and achieves the new state-of-the-art. |
Abstractive Text Summarization based on Language Model Conditioning and Locality Modeling (2020.lrec-1)
Copied to clipboard
Dmitrii Aksenov, Julian Moreno-Schneider, Peter Bourgonje, Robert Schwarzenberg, Leonhard Hennig, Georg Rehm
| Challenge: | Abstractive summarization is an NLP task with many real-world applications. |
| Approach: | They propose to use a pre-trained language model to train a Transformer-based neural model . they propose a new method of BERT-windowing to allow chunk-wise processing of texts longer than the BERT window size . |
| Outcome: | The proposed model outperforms baseline models on CNN/Daily Mail dataset and shows its superiority on German dataset. |
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents (N18-2)
Copied to clipboard
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, Nazli Goharian
| Challenge: | Existing abstractive summarization models focus on summarizing sentences and short documents. |
| Approach: | They propose a hierarchical encoder that models the discourse structure of a document, and an attentive discourse-aware decoder to generate the summary. |
| Outcome: | The proposed model significantly outperforms state-of-the-art models on two large-scale datasets of scientific papers. |
ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization is one of the areas influenced by pre-trained language models. |
| Approach: | They propose a Transformer-based encoder-decoder model pre-trained with three novel objectives to address this issue. |
| Outcome: | The proposed model outperforms previous models on six Persian summarization tasks . it also outperformed previous models in textual entailment, question paraphrasing, and question answering . |
Text Summarization with Pretrained Encoders (D19-1)
Copied to clipboard
| Challenge: | Existing pretraining languages such as ELMo and GPT have advanced a wide range of tasks. |
| Approach: | They propose a novel document-level encoder based on BERT which can express the semantics of a document and obtain representations for its sentences. |
| Outcome: | The proposed model achieves state-of-the-art in extractive and abstractive settings. |
ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that pre-trained text encoders can perform many NLP tasks with less resource. |
| Approach: | They propose a BERT-based Chinese text encoder enhanced by n-gram representations . they show reasonable performance when ZEN is trained on a small corpus . |
| Outcome: | The proposed encoder incorporates the comprehensive information of both the character sequence and words or phrases it contains. |
HierarchyNet: Learning to Summarize Source Code with Heterogeneous Representations (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing code summarization approaches ignore the interplay of dependencies among program elements and code hierarchy. |
| Approach: | They propose a code summarization approach utilizing Heterogeneous Code Representations (HCRs) and HierarchyNet. |
| Outcome: | The proposed method improves on existing models and pre-trained models. |
Multilingual Generation in Abstractive Summarization: A Comparative Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models for multilingual generation lack thorough analysis due to extensive linguistic diversity. |
| Approach: | They propose to classify multilingual generation methodologies into three categories based on their underlying modeling principles . they introduce an automatic metric to mitigate spurious correlations associated with language mixing . |
| Outcome: | The proposed model improves in high-resource, low-resourced, and zero-shot scenarios. |
CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing code LLMs adopt a specific architecture or rely on a unified encoder-decoder network for downstream tasks, lacking flexibility to operate in the optimal architecture for a particular task. |
| Approach: | They propose to initialize code LLMs with frozen off-the-shelf LLM and explore instruction-tuning to align with natural language instructions. |
| Outcome: | The proposed model outperforms open-source LLMs on 20 code-related benchmarks. |