An Imitation Learning Curriculum for Text Editing with Non-Autoregressive Models (2022.acl-long)
Copied to clipboard
| Challenge: | et al. (2017) show that imitation learning algorithms for machine translation introduce mismatches between training and inference that lead to undertraining and poor generalization in editing scenarios. |
| Approach: | They propose a framework for training non-autoregressive sequence-to-sequence models for editing tasks where the original input sequence is iteratively edited to produce the output. |
| Outcome: | The proposed framework significantly improves output quality and controls complexity better on the simplification task. |
Similar Papers
A Non-Autoregressive Edit-Based Approach to Controllable Text Simplification (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing models that generate generic simplified outputs for a given source text have been used to specify output properties. |
| Approach: | They propose a non-autoregressive model that iteratively edits an input sequence and incorporates lexical complexity information into the refinement process to generate simplifications that better match the desired output complexity. |
| Outcome: | The proposed model incorporates lexical complexity information into the refinement process to achieve more complex simplification operations such as content deletion and paraphrasing, as well as sentence splitting. |
Learning to Model Editing Processes (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing sequence generation models produce outputs in one pass, usually left-to-right . current models model only a single edit step, and do not fully model editing . |
| Approach: | They propose to model editing processes, modeling the whole process of iteratively generating sequences. |
| Outcome: | The proposed model improves performance on a variety of axes compared to previous models . iterative refinement and editing are central parts of human creative workflow . |
Quantifying Appropriateness of Summarization Data for Curriculum Learning (2021.eacl-main)
Copied to clipboard
| Challenge: | Summarization datasets are noisy, and summaries often do not reflect what is written in the source texts. |
| Approach: | They propose a method of curriculum learning to train summarization models from noisy data. |
| Outcome: | The proposed method improves the performance of pretrained and non-pretrained models on human evaluation. |
Reinforcement Learning based Curriculum Optimization for Neural Machine Translation (N19-1)
Copied to clipboard
| Challenge: | a heterogeneous training dataset can vary in characteristics such as domain, translation quality, and degree of difficulty. |
| Approach: | They propose to use reinforcement learning to learn an optimal curriculum for NMT training . they find it can beat uniform baselines and hand-designed, state-of-the-art curricula . |
| Outcome: | The proposed approach beats baselines and hand-designed curricula on English-to-French datasets. |
In-sample Curriculum Learning by Sequence Completion for Natural Language Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing work on curriculum learning rely on task-specific expertise and cannot generalize to different tasks. |
| Approach: | They propose to do in-sample curriculum learning for natural language generation tasks using human-crafted rules and a numeric score for each sample based on domain expertise to rank the model. |
| Outcome: | The proposed learning strategy generalizes well to different tasks and achieves significant improvements over baselines. |
Text Generation with Text-Editing Models (2022.naacl-tutorials)
Copied to clipboard
Eric Malmi, Yue Dong, Jonathan Mallinson, Aleksandr Chuklin, Jakub Adamek, Daniil Mirylenka, Felix Stahlberg, Sebastian Krause, Shankar Kumar, Aliaksei Severyn
| Challenge: | Text-editing models are a popular alternative to seq2seq for monolingual text generation tasks such as text summarization and style transfer. |
| Approach: | They propose to use text-editing models to predict edit operations applied to the source sequence and to generate outputs word-by-word from scratch. |
| Outcome: | This paper provides an overview of the text-edit based models and their current state-of-the-art approaches. |
Curriculum Learning for Domain Adaptation in Neural Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) performance drops when domains do not match and in-domain training data is scarce. |
| Approach: | They propose a curriculum learning approach to adapt generic neural machine translation models to a specific domain. |
| Outcome: | The proposed approach outperforms unadapted and adapted baselines in two domains and two language pairs. |
Text Editing as Imitation Game (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Text editing is an important domain of processing tasks to edit the text in a localized fashion, such as text simplification. |
| Approach: | They propose a nonautoregressive decoder for state-to-action demonstrations that parallels the decoding while retaining the dependencies between tokens. |
| Outcome: | The proposed model outperforms the autoregressive baselines on a suite of Arithmetic Equation benchmarks in terms of performance, efficiency, and robustness. |
Imitation Learning for Non-Autoregressive Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Existing non-autoregressive translation models lack parallel decoding, which is a bottleneck for NMT decoding. |
| Approach: | They propose a framework for non-autoregressive machine translation that emulates the autoregressive model by sampling sentence length in parallel. |
| Outcome: | The proposed model achieves 31.85 BLEU on WMT16 RoEn and 30.68 BLUE on IWSLT16 EnDe on the IWSLD16, WMT14 and WMT15 datasets. |
Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learning (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent advances in data-to-text generation have been focused on curriculum learning, which is a process of presenting training data in a specific order, starting from easy examples and moving on to more difficult ones, as the learner becomes more competent. |
| Approach: | They propose to use a curriculum learning process to change the order of training samples in a model based on the model's competence to improve model performance and convergence speed. |
| Outcome: | The proposed model shows faster convergence speed and reduced training time by 38.7% and performance by 4.84 BLEU. |