Proceedings of the 3rd Workshop on Neural Generation and Translation

Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems (D19-56)

Challenge:	The 3rd Workshop on Neural Machine Translation and Generation (WNGT) was held in concert with the annual conference of the Empirical Methods in Natural Language Processing (EMNLP 2019).
Approach:	They describe the results of the third workshop on Neural Generation and Translation held in concert with the annual conference of the Empirical Methods in Natural Language Processing (EMNLP 2019).
Outcome:	The results of the 3rd Workshop on Neural Machine Translation and Generation (WNGT) were summarized in Sections 3 and 4.

Copied to clipboard

Paweł Budzianowski, Ivan Vulić

Dialogue Systems & Conversational Agents

Challenge:	Statistical conversational systems are complex, timeintensive, expensive, and not easily transferable due to data scarcity.
Approach:	They propose a task-oriented dialogue model that operates on text input . they validate it on multi-domain task-orientated dialogues from a multi-word dataset .
Outcome:	The proposed model bypasses explicit policy and language generation modules on multi-domain task-oriented dialogues from the MultiWOZ dataset.

Recycling a Pre-trained BERT Encoder for Neural Machine Translation (D19-56)

Copied to clipboard

Kenji Imamura, Eiichiro Sumita

Generating a Common Question from Multiple Documents using Multi-source Encoder-Decoder Models (D19-56)

Challenge:	In monolingual tasks, the number of unlearned model parameters is as huge as the number learned parameters in the BERT model.
Approach:	They propose to apply a pre-trained Bidirectional Encoder Representations from Transformers (BERT) model to Transformer-based neural machine translation (NMT) based on the Transformer.
Outcome:	The proposed model is stable and efficient in low-resource settings.

Copied to clipboard

Woon Sang Cho, Yizhe Zhang, Sudha Rao, Chris Brockett, Sungjin Lee

Generating Diverse Story Continuations with Controllable Semantics (D19-56)

Challenge:	Ambiguous user queries can result in multiple topics being retrieved from search engines.
Approach:	They propose a task of generating a common question from multiple documents by training an RNN-based single encoder-decoder generator from document pairs and then a model that aggregates these word distributions to generate a question.
Outcome:	The proposed model significantly outperforms existing models when evaluated using automated metrics and human judgments on the MS-MARCO-QA dataset.

Copied to clipboard

Lifu Tu, Xiaoan Ding, Dong Yu, Kevin Gimpel

Domain Differential Adaptation for Neural Machine Translation (D19-56)

Challenge:	a new framework for controllable story continuation generation is proposed . we use frames to generate story continuations based on sentence attributes .
Approach:	They propose a framework for controlled generation of multiple, diverse outputs . they use sentiment, length, predicates, frames, and automatically-induced clusters as controllable dimensions .
Outcome:	The proposed model produces outputs that match target attributes, the authors show . it also yields higher metric scores than previous models, they show ."

Copied to clipboard

Zi-Yi Dou, Xinyi Wang, Junjie Hu, Graham Neubig

Machine Translation

Challenge:	Neural networks are data hungry and domain sensitive, so it is difficult to obtain labeled data for every domain.
Approach:	They propose a framework for domain adaptation where we model the difference between domains instead of smoothing over them.
Outcome:	The proposed framework improves on domain adaptation in multiple experimental settings.

Transformer-based Model for Single Documents Neural Summarization (D19-56)

Copied to clipboard

Elozino Egonmwan, Yllias Chali

Making Asynchronous Stochastic Gradient Descent Work for Transformers (D19-56)

Challenge:	Existing approaches for document summarization use manual feature engineering, integer linear programming and data-driven approaches.
Approach:	They propose a framework that encodes the source text first with a transformer, then a sequence-to-sequence model.
Outcome:	The proposed framework improves performance on extractive and abstractive document summarization task using the CNN/DailyMail and Newsroom datasets.

Copied to clipboard

Alham Fikri Aji, Kenneth Heafield

Controlled Text Generation for Data Augmentation in Intelligent Artificial Agents (D19-56)

Challenge:	Asynchronous stochastic gradient descent (SGD) converges poorly for Transformer models . synchronous SGD is faster at raw training speed since it avoids waiting for synchronization .
Approach:	They propose a method to restore convergence by summing several asynchronous updates instead of applying them immediately.
Outcome:	The proposed method achieves the same BLEU score 1.36 times faster than asynchronous SGD.

Copied to clipboard

Nikolaos Malandrakis, Minmin Shen, Anuj Goyal, Shuyang Gao, Abhishek Sethi, Angeliki Metallinou

Zero-Resource Neural Machine Translation with Monolingual Pivot Data (D19-56)

Challenge:	Data availability is a bottleneck during early stages of development of new capabilities for intelligent artificial agents.
Approach:	They propose to use conditional variational auto-encoders to augment training data of a popular commercial artificial agent with a small set of phrase templates to generate new semantically similar phrases.
Outcome:	The proposed approach outperforms the previous controlled text generation techniques with limited data and significantly outperformed the previous methods.

Copied to clipboard

Anna Currey, Kenneth Heafield

On the use of BERT for Neural Machine Translation (D19-56)

Challenge:	Neural machine translation systems have relied on large amounts of parallel training data between the source and target language.
Approach:	They propose methods for generating pseudo-parallel corpora using pivot-language data . they use English as the pivot language to train the zero-shot system .
Outcome:	The proposed methods improve the zero-shot neural machine translation system for a high-resource language pair using English as the pivot language.

Copied to clipboard

Stephane Clinchant, Kweon Woo Jung, Vassilina Nikoulina

On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation (D19-56)

Challenge:	Existing studies on using pretrained language models for supervised NMT have not been successful.
Approach:	They propose to integrate BERT pretrained models with supervised NMT models by using monolingual data.
Outcome:	The proposed models improve translation quality in English-German, English-Russian and IWSLT14 datasets.

Copied to clipboard

Victor Prokhorov, Ehsan Shareghi, Yingzhen Li, Mohammad Taher Pilehvar, Nigel Collier

Decomposing Textual Information For Style Transfer (D19-56)

Challenge:	Variational Autoencoders suffer from learning uninformative latent representations due to issues such as approximated posterior collapse or entanglement of the latent space.
Approach:	They propose to impose an explicit constraint on the Kullback-Leibler divergence term inside the VAE objective function to understand the significance of the KL term in controlling the information transmitted through the VAe channel.
Outcome:	The proposed constraint avoids posterior collapse, but it also controls the information transmitted through the VAE channel.

Copied to clipboard

Ivan P. Yamshchikov, Viacheslav Shibaev, Aleksander Nagaev, Jürgen Jost, Alexey Tikhonov

Unsupervised Evaluation Metrics and Learning Criteria for Non-Parallel Textual Transfer (D19-56)

Challenge:	Using a framework of style transfer for texts, we propose several empirical methods to assess information decomposition quality.
Approach:	They propose to use latent representations to effectively decompose different aspects of textual information using a framework of style transfer for texts.
Outcome:	The proposed methods show that higher quality representations correlate with higher performance in bilingual evaluation understudy (BLEU) between output and human-written reformulations.

Copied to clipboard

Richard Yuanzhe Pang, Kevin Gimpel

Responsible & Trustworthy NLP

Challenge:	Existing methods for textual transfer with no parallel corpora are insufficient to evaluate textual paraphrases with modified attributes or properties.
Approach:	They propose to add a metric for post-transfer classification accuracy and a method to combine them into a single overall score.
Outcome:	The proposed metrics correlate well with human judgments, at both the sentence-level and system-level.

Enhanced Transformer Model for Data-to-Text Generation (D19-56)

Copied to clipboard

Li Gong, Josep Crego, Jean Senellart

Generalization in Generation: A closer look at Exposure Bias (D19-56)

Challenge:	Neural models have shown significant progress on data-to-text generation tasks . data- to-text models generate descriptive texts from non-linguistic structured data .
Approach:	They propose a new data-to-text generation model which learns content selection and summary generation in an end-to end fashion.
Outcome:	The proposed model outperforms current state-of-the-art models on content selection precision and content ordering metrics.

Copied to clipboard

Florian Schmidt

Challenge:	Autoregressive generative models are often criticized for using ground-truth contexts at training time but generated ones at test time.
Approach:	They propose that generalization is the underlying property to address and propose unconditional generation as its fundamental benchmark.
Outcome:	The proposed model is generalized and can handle true and generated contexts.

Machine Translation of Restaurant Reviews: New Corpus for Domain Adaptation and Robustness (D19-56)

Copied to clipboard

Alexandre Berard, Ioan Calapodescu, Marc Dymetman, Claude Roux, Jean-Luc Meunier, Vassilina Nikoulina

Responsible & Trustworthy NLP

Challenge:	BLEU: MT is a very robust and efficient way to translate user-generated content.
Approach:	They propose a task to encourage research on MT robustness and domain adaptation . they ask professionals to translate 11.5k french 4SQ reviews to English .
Outcome:	The proposed task improves on the existing MT systems in a real-world scenario . the proposed methods improve translation accuracy and sentiment analysis .

Adaptively Scheduled Multitask Learning: The Case of Low-Resource Neural Machine Translation (D19-56)

Copied to clipboard

Poorya Zaremoodi, Gholamreza Haffari

On the Importance of Word Boundaries in Character-level Neural Machine Translation (D19-56)

Challenge:	Neural Machine Translation suffers from the lack of bilingual data in low-resource scenarios.
Approach:	They propose to inject inductive biases into Neural Machine Translation (NMT) using auxiliary syntactic and semantic tasks.
Outcome:	The proposed approach improves translation quality by reweighing training data of main and auxiliary tasks based on their contributions to generalisability of main task.

Copied to clipboard

Duygu Ataman, Orhan Firat, Mattia A. Di Gangi, Marcello Federico, Alexandra Birch

Big Bidirectional Insertion Representations for Documents (D19-56)

Challenge:	Neural Machine Translation models typically use a fixed-size lexical vocabulary . subword segmentation methods rely on statistical heuristics that lack any linguistic notion .
Approach:	They propose a hierarchical decoding architecture for character-level NMT using subwords . they propose fewer parameters and a more efficient approach to perform translation at the level of words .
Outcome:	The proposed model can reach higher translation accuracy than the subword-level model with fewer parameters while maintaining longer-distance contextual and grammatical dependencies.

Copied to clipboard

Lala Li, William Chan

Semantic Text Processing

Challenge:	Recent studies suggest we are nearing human-level parity for sentence-level translation in certain domains.
Approach:	They propose an insertion-based model for document-level translation tasks that embeds sentence alignment between the source and target document.
Outcome:	The proposed model improves on the WMT’19 English->German translation task by +4.3 BLEU compared with the Insertion Transformer baseline.

A Margin-based Loss with Synthetic Negative Samples for Continuous-output Machine Translation (D19-56)

Copied to clipboard

Gayatri Bhat, Sachin Kumar, Yulia Tsvetkov

Mixed Multi-Head Self-Attention for Neural Machine Translation (D19-56)

Challenge:	Existing methods for word embeddings generate faster training with fewer learnable parameters.
Approach:	They propose a novel margin-based loss that uses only predicted and target embeddings . they argue that the loss is more consistent and interpretable than other margin--based losses .
Outcome:	The proposed model is more consistent and interpretable than other margin-based losses.

Copied to clipboard

Hongyi Cui, Shohei Iida, Po-Hsuan Hung, Takehito Utsuro, Masaaki Nagata

Paraphrasing with Large Language Models (D19-56)

Challenge:	Recent advances in neural machine translation have been made in the field of multi-head self-attention and there is no explicit mechanism to ensure that different attention heads capture different features.
Approach:	They propose a novel multi-head self-attention model which models not only global and local attention but also forward and backward attention in different attention heads.
Outcome:	The proposed model improves on WAT17 English-Japanese and IWSLT14 German-English translation tasks without increasing the number of parameters.

Copied to clipboard

Sam Witteveen, Martin Andrews

Interrogating the Explanatory Power of Attention in Neural Machine Translation (D19-56)

Challenge:	Recent work has shown large language models are adept at text generation and fine-tuning for downstream NLP tasks.
Approach:	They propose a system that generates paraphrased examples in autoregressive fashion using a neural network without the need for techniques such as top-k word selection or beam search.
Outcome:	The proposed system generates paraphrased examples in autoregressive fashion without the need for techniques such as top-k word selection or beam search.

Copied to clipboard

Pooya Moradi, Nishant Kambhatla, Anoop Sarkar

Machine Translation

Challenge:	Attention models are often used to justify the model’s decision in generating a token but it has not been rigorously established to what extent attention is a reliable source of information in NMT.
Approach:	They propose to use attention models to modify crucial aspects of the trained attention model to produce function and content words in the translation process.
Outcome:	The proposed models preserve function and content words in the translation process compared to state-of-the-art models.

Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and Performance for Low-Resource Machine Translation (D19-56)

Copied to clipboard

Kenton Murray, Jeffery Kinnison, Toan Q. Nguyen, Walter Scheirer, David Chiang

Responsible & Trustworthy NLP

Challenge:	Neural sequence-to-sequence models are sensitive to architecture and hyperparameter settings.
Approach:	They incorporate architecture search into a single training run through auto-sizing . they show that auto-size can improve BLEU scores by up to 3.9 points .
Outcome:	The proposed algorithm improves BLEU scores on low-resource language pairs while removing one-third of the parameters from the model.

Learning to Generate Word- and Phrase-Embeddings for Efficient Phrase-Based Neural Machine Translation (D19-56)

Copied to clipboard

Chan Young Park, Yulia Tsvetkov

Semantic Text Processing

Challenge:	Neural machine translation (NMT) often fails in one-to-many translation, e.g., in the translation of multi-word expressions, compounds, and collocations.
Approach:	They propose a phrase-based NMT model that generates embeddings of words or phrases.
Outcome:	The proposed model performs on par with state-of-the-art phrase-based NMT.

Transformer and seq2seq model for Paraphrase Generation (D19-56)

Copied to clipboard

Elozino Egonmwan, Yllias Chali

Monash University’s Submissions to the WNGT 2019 Document Translation Task (D19-56)

Challenge:	Existing methods for generating paraphrases fall into one of these broad categories -rule-based, seq2seq, deep generative models and a varied combination.
Approach:	They propose a framework that combines transformer and sequence-to-sequence models for better quality of generated paraphrases.
Outcome:	The proposed framework improves on two datasets-QUORA and MSCOCO using transformer and sequence-to-sequence models.

Copied to clipboard

Sameen Maruf, Gholamreza Haffari

SYSTRAN @ WNGT 2019: DGT Task (D19-56)

Challenge:	Despite the boom of work on document-level machine translation in the past two years, there has been a lack of the application of the proposed approaches to MT shared tasks.
Approach:	They propose to employ an established document-level neural machine translation model for the shared task of Rotowire document translation organised by the 3rd Workshop on Neural Generation and Translation (WNGT 2019).
Outcome:	The proposed model achieves a BLEU score of 39.83 for En-De and 45.06 for De-En translation directions on the Rotowire test set.

Copied to clipboard

Li Gong, Josep Crego, Jean Senellart

Challenge:	SYSTRAN participates in Document-level generation and trans-lation (DGT) task . data-to-text generation tasks are difficult because of the content selection and text generation data.
Approach:	They propose a Transformer-based datato-text generation model which jointly learns content selection and text generation.
Outcome:	The proposed model outperforms current state-of-the-art system on BLEU, content selection precision and content ordering metics.

University of Edinburgh’s submission to the Document-level Generation and Translation Shared Task (D19-56)

Copied to clipboard

Ratish Puduppully, Jonathan Mallinson, Mirella Lapata

Naver Labs Europe’s Systems for the Document-Level Generation and Translation Task at WNGT 2019 (D19-56)

Challenge:	University of Edinburgh participated in all six tracks: NLG, MT, and MT+NLG with English and German as targeted languages.
Approach:	The University of Edinburgh participated in all six tracks: NLG, MT, and MT+NLG . they submitted a multilingual system based on the Content Selection and Planning model .
Outcome:	The University of Edinburgh participated in all six tracks with English and German as target languages.

Copied to clipboard

Fahimeh Saleh, Alexandre Berard, Ioan Calapodescu, Laurent Besacier

From Research to Production and Back: Ludicrously Fast Neural Machine Translation (D19-56)

Challenge:	Recent advances in machine translation and natural language generation have created many challenges in this field especially when context is considered.
Approach:	They propose to leverage data from machine translation and natural language generation tasks to do transfer learning between MT, NLG and MT with source-side metadata.
Outcome:	The proposed approach outperforms the previous state-of-the-art on the Rotowire NLG task.

Copied to clipboard

Young Jin Kim, Marcin Junczys-Dowmunt, Hany Hassan, Alham Fikri Aji, Kenneth Heafield, Roman Grundkiewicz, Nikolay Bogoychev

Machine Translation

Challenge:	Using the dominating submissions to the previous edition of the shared task, we develop improved teacher-student training via multi-agent dual-learning and noisy backward-forward translation for Transformer-based student models.
Approach:	They propose to use multi-agent dual-learning and noisy backward-forward translation to improve teacher-student training for Transformer-based student models.
Outcome:	The proposed model outperforms submissions to the previous edition of the WNGT efficiency shared task by 4 BLEU points and 10 BLUE points respectively.

Selecting, Planning, and Rewriting: A Modular Approach for Data-to-Document Generation and Translation (D19-56)

Copied to clipboard

Lesly Miculicich, Marc Marone, Hany Hassan