Seq2Edits: Sequence Transduction Using Span-level Edit Operations

Challenge:	Seq2Edits is an open-vocabulary approach to sequence editing for natural language processing tasks with a high degree of overlap between input and output texts.
Approach:	They propose an open-vocabulary approach to sequence editing for NLP tasks with a high degree of overlap between input and output texts.
Outcome:	The proposed approach speeds up inference by up to 5.2x compared to full sequence models . it improves explainability by associating each edit operation with a human-readable tag.

Similar Papers

Overview

Challenge

Approach

Outcome

Text Generation with Text-Editing Models (2022.naacl-tutorials)

Copied to clipboard

Eric Malmi, Yue Dong, Jonathan Mallinson, Aleksandr Chuklin, Jakub Adamek, Daniil Mirylenka, Felix Stahlberg, Sebastian Krause, Shankar Kumar, Aliaksei Severyn

Language Models

Challenge:	Text-editing models are a popular alternative to seq2seq for monolingual text generation tasks such as text summarization and style transfer.
Approach:	They propose to use text-editing models to predict edit operations applied to the source sequence and to generate outputs word-by-word from scratch.
Outcome:	This paper provides an overview of the text-edit based models and their current state-of-the-art approaches.

Reducing Sequence Length by Predicting Edit Spans with Large Language Models (2023.emnlp-main)

Copied to clipboard

Masahiro Kaneko, Naoaki Okazaki

Language Models

Challenge:	Large Language Models (LLMs) have demonstrated remarkable performance in various tasks and gained significant attention.
Approach:	They propose to predict edit spans for local sequence transduction tasks by predicting edit span with a position of the source text and corrected tokens.
Outcome:	The proposed method reduces the length of the target sequence and the computational cost for inference by as small as 21%.

Multi-pass Decoding for Grammatical Error Correction (2024.emnlp-main)

Copied to clipboard

Xiaoying Wang, Lingling Mu, Jingyi Zhang, Hongfei Xu

Language Models

Challenge:	Seq2edit models decode only once without aware of subsequent tokens.
Approach:	They propose to iteratively refine the correction results of seq2seq models via Multi-Pass Decoding (MPD) to improve performance, but MPD increases inference costs . they propose to merge the source input and previous round correction result into one sequence.
Outcome:	Experiments on the CoNLL-14 and BEA-19 test set show that the proposed approach improves over baselines.

Transforming Sequence Tagging Into A Seq2Seq Task (2022.emnlp-main)

Copied to clipboard

Karthik Raman, Iftekhar Naim, Jiecao Chen, Kazuma Hashimoto, Kiran Yalasangi, Krishna Srinivasan

Syntactic Text Processing

Challenge:	Pretrained, large, generative language models have had great success in a wide range of sequence tagging and structured prediction tasks.
Approach:	They propose to use a new format for casting input text sentences and their output labels into the input and target of a Seq2Seq model and introduce it to test their hypothesis.
Outcome:	The proposed format shows to be both simpler and more effective and devoid of hallucination.

Span-based Semantic Parsing for Compositional Generalization (2021.acl-long)

Copied to clipboard

Jonathan Herzig, Jonathan Berant

Semantic Text Processing

Challenge:	despite success of sequence-to-sequence models, they fail in compositional generalization . a span-based parser that predicts a utterance over spans improves performance .
Approach:	They propose a span-based parser that predicts a utterance over a given span tree . they propose to use CKY to encode how partial programs compose over spans .
Outcome:	The proposed model performs better on random splits than baselines that require compositional generalization.

Automatic Grammatical Error Correction for Sequence-to-sequence Text Generation: An Empirical Study (P19-1)

Copied to clipboard

Tao Ge, Xingxing Zhang, Furu Wei, Ming Zhou

Language Models

Challenge:	Sequence-to-sequence (seq2sequ) models have a weakness: they cannot always generate sentences without grammatical errors.
Approach:	They propose to use automatic grammatical error correction to improve seq2seq models . they conduct experiments on machine translation, formality style transfer, sentence compression and simplification .
Outcome:	The proposed system can improve grammaticality of generated text and improve formal style tasks.

FiD-Ex: Improving Sequence-to-Sequence Models for Extractive Rationale Generation (2021.emnlp-main)

Copied to clipboard

Kushal Lakhotia, Bhargavi Paranjape, Asish Ghoshal, Scott Yih, Yashar Mehdad, Srini Iyer

Responsible & Trustworthy NLP

Challenge:	Pre-trained sequence to sequence models are effective in making and generating NL explanations, but they have many shortcomings.
Approach:	They propose a model that uses sentence markers to eliminate explanation fabrication . they use fusion-in-decoder architecture to handle long input contexts .
Outcome:	The proposed model significantly improves on the ERASER explainability benchmark.

EdiT5: Semi-Autoregressive Text Editing with T5 Warm-Start (2022.findings-emnlp)

Copied to clipboard

Jonathan Mallinson, Jakub Adamek, Eric Malmi, Aliaksei Severyn

Syntactic Text Processing

Challenge:	Pre-trained seq2seq models have established strong baselines for text-to-text transduction tasks.
Approach:	They propose a semi-autoregressive text-editing approach that combines the strengths of non-auto-regressively text- editing and autoregressive decoding.
Outcome:	The proposed model is faster at inference times than conventional models while being capable of modeling flexible input-output transformations.

Text Editing as Imitation Game (2022.findings-emnlp)

Copied to clipboard

Ning Shi, Bin Tang, Bo Yuan, Longtao Huang, Yewen Pu, Jie Fu, Zhouhan Lin

Syntactic Text Processing

Challenge:	Text editing is an important domain of processing tasks to edit the text in a localized fashion, such as text simplification.
Approach:	They propose a nonautoregressive decoder for state-to-action demonstrations that parallels the decoding while retaining the dependencies between tokens.
Outcome:	The proposed model outperforms the autoregressive baselines on a suite of Arithmetic Equation benchmarks in terms of performance, efficiency, and robustness.

FELIX: Flexible Text Editing Through Tagging and Insertion (2020.findings-emnlp)

Copied to clipboard

Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, Guillermo Garrido

Syntactic Text Processing

Challenge:	FELIX is efficient in low-resource settings and fast at inference time, while being capable of modeling flexible input-output transformations.
Approach:	They propose a flexible text-editing approach that decomposes a text-generating task into two sub-tasks: tagging and insertion.
Outcome:	The proposed model is efficient in low-resource settings and fast at inference time while being capable of modeling flexible input-output transformations.