Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing strategies to teach pre-trained models to generate simple texts are inadequate. |
| Approach: | They propose a continued pre-training strategy to teach pre-trained models to generate simple texts by randomly masking text spans in ordinary texts. |
| Outcome: | The proposed strategy improves on lexical simplification, sentence simplification and document-level simplification tasks over existing models. |
Similar Papers
Complicate Then Simplify: A Novel Way to Explore Pre-trained Models for Text Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing frameworks for text classification employing pre-trained models are constrained by the difficulty of the task. |
| Approach: | They propose a framework which implements a two-stage training strategy to fully exploit the knowledge in pre-trained models. |
| Outcome: | The proposed framework outperforms state-of-the-art classification models on six text classification corpora. |
Controlling Pre-trained Language Models for Grade-Specific Text Simplification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to text simplification control output complexity at corpus level disregarding complexity of individual inputs and considering only one level of output complexity. |
| Approach: | They propose a method that predicts edit operations required for a specific grade level . they say this approach improves the quality of the simplified outputs over corpus-level heuristics . |
| Outcome: | The proposed method improves the readability of simplified outputs over corpus-level search-based heuristics. |
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020.acl-main)
Copied to clipboard
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer
| Challenge: | Recent work has shown gains by improving the distribution of masked tokens and the order in which mucked tokens are predicted. |
| Approach: | They propose a denoising autoencoder for pretraining sequence-to-sequence models that corrupts text with an arbitrary noising function and learns a model to reconstruct the original text. |
| Outcome: | The proposed model outperforms RoBERTa on GLUE and SQUAD and provides a 1.1 BLEU increase over a back-translation system for machine translation. |
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that our binary masked language models encode information necessary for solving downstream tasks. |
| Approach: | They propose an efficient method of utilizing pretrained language models where selective binary masks are learned instead of finetuning. |
| Outcome: | Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that the proposed method yields comparable performance to finetuning, but has a much smaller memory footprint when multiple tasks need to be solved. |
Self-supervised Graph Masking Pre-training for Graph-to-Text Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models (PLMs) have advanced Graph-to-Text generation by processing the linearised version of a graph. |
| Approach: | They propose to mask pre-training tasks that neither require supervision signals nor adjust the architecture of the underlying pre-trained encoder-decoder model. |
| Outcome: | The proposed method achieves state-of-the-art results on WebNLG+2020 and EventNarrative datasets and is very efficient in the low-resource setting. |
Language Models for German Text Simplification: Overcoming Parallel Data Scarcity through Style-specific Pre-training (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train automatic text simplification systems for languages other than English are limited by the lack of parallel data. |
| Approach: | They propose to use German Easy Language as a corpus of automatic text simplification systems to fine-tune language models to the style characteristics of the language. |
| Outcome: | The proposed language models adapt to the style characteristics of Easy Language and output more accessible texts. |
Explainable Prediction of Text Complexity: The Missing Preliminaries for Text Simplification (2021.acl-long)
Copied to clipboard
| Challenge: | Text simplification reduces the language complexity of professional content for accessibility purposes. |
| Approach: | They propose that text simplification can be decomposed into a pipeline of tasks . they show that the pipeline can be used to predict whether a text needs to be simplified . |
| Outcome: | The proposed model improves the performance of out-of-sample simplification tests on a blackbox lexical model . the proposed model reduces the complexity of professional text by a large margin . |
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2020.tacl-1)
Copied to clipboard
| Challenge: | Pre-training methods like BERT mask individual words or subword units, but many tasks involve reasoning about relationships between two or more spans of text. |
| Approach: | They propose a pre-training method that masks contiguous random spans instead of random tokens to train the span boundary representations to predict the entire content of the masked span. |
| Outcome: | The proposed method outperforms BERT and its better-tuned baselines on span selection tasks and on coreference resolution tasks. |
Text-to-Code Generation with Modality-relative Pre-training (2024.eacl-long)
Copied to clipboard
| Challenge: | Large pre-trained language models have been applied to programming language tasks with great success, often through further pre-training of a strictly-natural language model. |
| Approach: | They propose to map programming language modalities into the same embedding space by separating embeddable spaces between modality and modality-relative training objectives. |
| Outcome: | The proposed model can be adapted and represented differently depending on which modality they belong to and to the ultimate benefit of the downstream task. |
Learning Rich Representation of Keyphrases from Text (2022.findings-naacl)
Copied to clipboard
| Challenge: | Prior work has referred to extractive (part of document) or abstractive (not part of document). |
| Approach: | They propose to use a new pre-training objective to introduce keyphrases into transformer language models in discriminative and generative settings. |
| Outcome: | The proposed model improves performance in discriminative and generative settings and also improves on named entity recognition, question answering, relation extraction and abstractive summarization tasks. |