Learning to Stop in Structured Prediction for Neural Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Beam search optimization solves many problems in neural machine translation, but lacks principled stopping criteria and does not learn how to stop during training. |
| Approach: | They propose a ranking method which enables an optimal beam search stop-ping criteria and a structured prediction loss function which penalizes suboptimal finished candidates produced by beam search during training. |
| Outcome: | Experiments on synthetic and real languages show that the proposed methods improve translation quality and length. |
Similar Papers
Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Beam search is widely used in neural machine translation, but beam sizes larger than 5 hurt translation quality. |
| Approach: | They propose to use beam search to improve translation quality by using hyperparameter-free methods that outperform the widely-used heuristic of length normalization by +2.0 BLEU. |
| Outcome: | The proposed methods outperform the widely-used heuristic on Chinese-to-English translation and achieve the best results among all methods. |
Improving Beam Search by Removing Monotonic Constraint for Neural Machine Translation (P18-2)
Copied to clipboard
| Challenge: | a beam search algorithm produces monotonic left-to-right order, meaning a hypothesis cannot be revisited . a proposed algorithm allows discarded hypotheses to be recovered in a later step. |
| Approach: | They propose to decode a beam search algorithm that considers multiple hypotheses simultaneously . they propose to maintain all found hypothese a single priority queue and a universal score function . |
| Outcome: | The proposed algorithm improves translations even for high-performance models in English-Japanese translation task. |
Machine Translation Decoding beyond Beam Search (2021.emnlp-main)
Copied to clipboard
Rémi Leblond, Jean-Baptiste Alayrac, Laurent Sifre, Miruna Pislar, Lespiau Jean-Baptiste, Ioannis Antonoglou, Karen Simonyan, Oriol Vinyals
| Challenge: | a new study examines whether beam search can be replaced by a more powerful metric-driven search technique. |
| Approach: | They propose a beam search method which is agnostic to the end metric and report results on a variety of metrics. |
| Outcome: | The proposed method is based on a Monte-Carlo Tree Search (MCTS) based method and shows it can be used in language applications. |
A Simple and Effective Approach to Coverage-Aware Neural Machine Translation (P18-2)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) models are used to solve translation problems using long-term models. |
| Approach: | They propose a method to seek a better balance between model confidence and length preference for Neural Machine Translation. |
| Outcome: | The proposed model improves on Chinese-English and English-German translation tasks. |
Discriminative Reranking for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | reranking models allow the integration of rich features to select a better output hypothesis within an n-best list or lattice. |
| Approach: | They use discriminative reranking to train a large transformer architecture to train an ranked list of hypotheses. |
| Outcome: | Experiments on four WMT directions show that discriminative reranking improves translation quality. |
Restricted or Not: A General Training Framework for Neural Machine Translation (2022.acl-srw)
Copied to clipboard
| Challenge: | Existing work imposes constraints on beam search decoding, which limits the concurrent processing ability of the model in deployment. |
| Approach: | They propose a general training framework that allows a model to support both restricted and unrestricted translations by adopting an additional auxiliary training process without constraining the decoding process. |
| Outcome: | The proposed training framework is tested on simulated and original benchmarks. |
Incremental Beam Manipulation for Natural Language Generation (2021.eacl-main)
Copied to clipboard
| Challenge: | a larger beam size can lead to deteriorating performance of natural language generation systems due to model errors . performance of NLG systems can plateau or even decrease when beam sizes larger than 10 are used . |
| Approach: | They propose to rerank the output of beam search to produce a good set of hypotheses . they propose incremental beam manipulation to discarded hypothese . |
| Outcome: | The proposed method outperforms a strong reranker on the E2E and WebNLG datasets while being on par with the existing method. |
Classical Structured Prediction Losses for Sequence to Sequence Learning (N18-1)
Copied to clipboard
| Challenge: | Recent work on training neural attention models at the sequence level has focused on a series of objective functions commonly used for structured prediction. |
| Approach: | They propose to use objective functions commonly used to train linear models for structured prediction to train neural attention models at the sequence-level using either reinforcement learning-style methods or beam search optimization. |
| Outcome: | The proposed model outperforms beam search optimization on German-English translation and abstractive summarization tasks. |
A Call for Clarity in Beam Search: How It Works and When It Stops (2024.lrec-main)
Copied to clipboard
| Challenge: | Empirical results show that a modified beam decoding implementation improves decoding performance of strong, neural language generation models. |
| Approach: | They propose a modification to a beam decoding implementation that generalizes the stopping criterion and provides flexibility to the depth of search. |
| Outcome: | The proposed method improves decoding performance of strong models on news text summarization and machine translation over diverse language pairs with negligible inference slowdown. |
Checkpoint Reranking: An Approach to Select Better Hypothesis for Neural Machine Translation Systems (2020.acl-srw)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) has produced excellent results in the field of machine translation due to generation of high-quality translations for different language pairs. |
| Approach: | They propose a method of re-ranking the outputs of Neural Machine Translation systems by focusing on the decoder's ability to generate distinct tokens and without the use of any language model or data. |
| Outcome: | The proposed method achieves translation improvement up to +0.16 BLEU points over baseline. |