Papers by Kenneth Heafield
Parallel Sentence Mining by Constrained Decoding (2020.acl-main)
Copied to clipboard
| Challenge: | Using a neural machine translation system to extract parallel sentences from two monolingual corpora is challenging. |
| Approach: | They propose a method to extract parallel sentences from two monolingual corpora by constraining the decoding by a prefix tree built on the other corpus. |
| Outcome: | The proposed method achieves comparable results to other submissions on the BUCC shared task. |
ParaCrawl: Web-Scale Acquisition of Parallel Corpora (2020.acl-main)
Copied to clipboard
Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ramírez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, Jaume Zaragoza
| Challenge: | We describe methods to create the largest publicly available parallel corpora by crawling the web . parallel corpus is essential for building highquality machine translation systems . |
| Approach: | They describe methods to create largest publicly available parallel corpora by crawling web sites . they empirically compare alternative methods and publish benchmark data sets . |
| Outcome: | The proposed methods improve state-of-the-art results on common benchmarks, the authors show . the pipeline has been tested on Russian, Sinhala, Nepali, Tagalog, Swahili, and Somali . |
Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training (D19-1)
Copied to clipboard
| Challenge: | In recent years, neural network models have grown dramatically in terms of number of parameters, so exchanging gradients during data-parallel training is costly in terms both of bandwidth and time. |
| Approach: | They propose to combine the compressed global gradient with the local gradient to restore Transformer convergence while RNNs converge faster. |
| Outcome: | The proposed method restores transformer convergence while RNNs converge faster. |
In Neural Machine Translation, What Does Transfer Learning Transfer? (2020.acl-main)
Copied to clipboard
| Challenge: | a recent study found that word embeddings are not necessary for transfer learning. |
| Approach: | They perform several ablation studies that limit information transfer and measure the quality impact across three language pairs to gain a black-box understanding of transfer learning. |
| Outcome: | The proposed method can eliminate the need for a warm-up phase when training transformer models in high resource language pairs. |
TranslateLocally: Blazing-fast translation running on the local CPU (2021.emnlp-demo)
Copied to clipboard
| Challenge: | Using cloud-based translation providers carries privacy risks, as users lose control of their data once it enters the web. |
| Approach: | They propose a desktop translation application that runs locally on a user's desktop or laptop CPU. translateLocally delivers cloud-like translation speed and quality even on 10 year old hardware. |
| Outcome: | The open-source translation system runs on Linux, Windows and macOS on desktops and laptops. |
Cheat Codes to Quantify Missing Source Information in Neural Machine Translation (2022.naacl-main)
Copied to clipboard
| Challenge: | a neural machine translation system generates a translation t in the target language, but for any sentence of non-trivial complexity, the translation s is not unique. |
| Approach: | They propose a method to quantify the amount of information missing in a machine translation system. |
| Outcome: | The proposed model captures extra information from a single float representation of the target sentence and reproduces it with two 32-bit floats per target token. |
The EuroPat Corpus: A Parallel Corpus of European Patent Data (2022.lrec-1)
Copied to clipboard
| Challenge: | a new corpus of patent-specific parallel data is available for 6 official European languages paired with English: German, Spanish, French, Croatian, Norwegian, and Polish. |
| Approach: | They present a patent-specific corpus of parallel data for 6 official European languages paired with English: German, Spanish, French, Croatian, Norwegian, and Polish. |
| Outcome: | The filtered corpus ranges in size from 51 million sentences (Spanish-English) to 154k sentences (Croatian-English), with the unfiltered (raw) corpus being up to 2 times larger. |
From Research to Production and Back: Ludicrously Fast Neural Machine Translation (D19-56)
Copied to clipboard
Young Jin Kim, Marcin Junczys-Dowmunt, Hany Hassan, Alham Fikri Aji, Kenneth Heafield, Roman Grundkiewicz, Nikolay Bogoychev
| Challenge: | Using the dominating submissions to the previous edition of the shared task, we develop improved teacher-student training via multi-agent dual-learning and noisy backward-forward translation for Transformer-based student models. |
| Approach: | They propose to use multi-agent dual-learning and noisy backward-forward translation to improve teacher-student training for Transformer-based student models. |
| Outcome: | The proposed model outperforms submissions to the previous edition of the WNGT efficiency shared task by 4 BLEU points and 10 BLUE points respectively. |
Efficient Methods for Natural Language Processing: A Survey (2023.tacl-1)
Copied to clipboard
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
| Challenge: | Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data, but using only scale to improve performance means resource consumption also grows. |
| Approach: | They propose to use data, time, storage, or energy to improve model performance. |
| Outcome: | The proposed methods and findings provide guidance for conducting NLP under limited resources and point towards promising research directions for developing more efficient methods. |
An Open Dataset and Model for Language Identification (2023.acl-short)
Copied to clipboard
| Challenge: | Existing LID systems perform poorly on low-resource languages, causing 'representation washing', where the community is given a false view of the actual progress of low-source NLP. |
| Approach: | They propose a model which achieves a macro-average F1 score of 0.93 and a false positive rate of 0.033% across 201 languages, outperforming previous work. |
| Outcome: | The proposed model outperforms existing models and datasets on 201 languages and a false positive rate of 0.033%. |
Gender bias amplification during Speed-Quality optimization in Neural Machine Translation (2021.acl-short)
Copied to clipboard
| Challenge: | et al., 2002) show that gendered noun translation performance degrades faster than BLEU. |
| Approach: | They propose to use greedy search, quantization, AANs and shallow decoders to speed up decoding . they find minimal degradation of BLEU, but gendered noun translation degrades faster . |
| Outcome: | The proposed model degrades gendered noun translation performance faster than other models. |
Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca (2024.findings-eacl)
Copied to clipboard
| Challenge: | Foundational large language models (LLMs) can be instruction-tuned to perform open-domain question answering, facilitating applications like chat assistants. |
| Approach: | They employ a dataset and machine translations of it to form multilingual data and use it to tune LLMs. |
| Outcome: | The proposed model is on par or better than a model for each language, and multilingual tuning with downsampled data is as powerful and robust. |
Losing Heads in the Lottery: Pruning Transformer Attention in Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent research shows that attention heads are not confident in their decisions and can be pruned. |
| Approach: | They apply the lottery ticket hypothesis to prune heads in early training . they find that the pruned model is 1.5 times faster at inference . |
| Outcome: | The proposed method is 1.5 times faster at inference, but at the cost of longer training. |
Document-Level Machine Translation with Large-Scale Public Parallel Corpora (2024.acl-long)
Copied to clipboard
| Challenge: | Document-level machine translation has inherent advantages over sentence-level translation due to additional information available to a model from document context. |
| Approach: | They propose to use document context to train context-aware models on these datasets and to use it to model document-level phenomena. |
| Outcome: | The proposed models improve translation quality and target document-level phenomena by incorporating contextual information from several preceding sentences. |
Making Asynchronous Stochastic Gradient Descent Work for Transformers (D19-56)
Copied to clipboard
| Challenge: | Asynchronous stochastic gradient descent (SGD) converges poorly for Transformer models . synchronous SGD is faster at raw training speed since it avoids waiting for synchronization . |
| Approach: | They propose a method to restore convergence by summing several asynchronous updates instead of applying them immediately. |
| Outcome: | The proposed method achieves the same BLEU score 1.36 times faster than asynchronous SGD. |
Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | In order to achieve faster training we increase the mini-batch size and scale the learning rate accordingly. |
| Approach: | They propose a technique that delays gradient updates by increasing the mini-batch size to improve the model's convergence. |
| Outcome: | The proposed technique can train a shallow machine translation system 27% faster than an optimized baseline with negligible penalty in BLEU. |
Marian: Fast Neural Machine Translation in C++ (P18-4)
Copied to clipboard
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, Alexandra Birch
| Challenge: | In this paper, we present Marian, an efficient and self-contained Neural Machine Translation framework . Marian is written in pure C++ with minimal dependencies . |
| Approach: | They present Marian, an efficient and self-contained Neural Machine Translation framework written in pure C++ with minimal dependencies. |
| Outcome: | The proposed framework achieves high training and translation speed with minimal dependencies . it is currently being deployed in multiple European projects . |
Zero-Resource Neural Machine Translation with Monolingual Pivot Data (D19-56)
Copied to clipboard
| Challenge: | Neural machine translation systems have relied on large amounts of parallel training data between the source and target language. |
| Approach: | They propose methods for generating pseudo-parallel corpora using pivot-language data . they use English as the pivot language to train the zero-shot system . |
| Outcome: | The proposed methods improve the zero-shot neural machine translation system for a high-resource language pair using English as the pivot language. |
Constrained Regeneration for Cross-Lingual Query-Focused Extractive Summarization (2022.coling-1)
Copied to clipboard
Elsbeth Turcan, David Wan, Faisal Ladhak, Petra Galuscakova, Sukanta Sen, Svetlana Tchistiakova, Weijia Xu, Marine Carpuat, Kenneth Heafield, Douglas Oard, Kathleen McKeown
| Challenge: | Query-focused summarization of foreign-language documents can help a user understand whether a document is relevant to a query term. |
| Approach: | They propose to use machine translation and post-editing to improve human relevance judgments . they include a query term in a summary when its translation appears in the source document . |
| Outcome: | The proposed approach improves human relevance judgments by including a query term in a summary when its translation appears in the source document. |
Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task (N18-1)
Copied to clipboard
| Challenge: | Previously, neural methods in grammatical error correction did not reach state-of-the-art results compared to phrase-based statistical machine translation (SMT) systems that improve on results by SMT use their set-up as a backbone for more complex systems. |
| Approach: | They propose a set of model-independent methods for neural GEC that can be easily applied in most GEC settings. |
| Outcome: | The proposed methods outperform state-of-the-art neural GEC systems by 10% M2 on the CoNLL-2014 benchmark and 5.9% on the JFLEG test set. |
Code-Switched Language Identification is Harder Than You Think (2024.eacl-long)
Copied to clipboard
| Challenge: | Code switching (CS) is a common phenomenon in written and spoken communication, but is handled poorly by many NLP applications. |
| Approach: | They propose to use CS language identification for corpus building to make it more realistic by scaling it to more languages and considering models with simpler architectures for faster inference. |
| Outcome: | The proposed system is based on a sentence-level multi-label tagging problem and provides recommendations for future work. |
Cheating to Identify Hard Problems for Neural Machine Translation (2023.findings-eacl)
Copied to clipboard
| Challenge: | We find that the most frequent tokens are not necessarily the most accurately translated due to these often being function words and punctuation that can be used more flexibly in translation. |
| Approach: | They propose a method to provide a compressed representation of the target as an input and a second method to fine-tune a standard transformer model. |
| Outcome: | The proposed method produces models with higher scores, but with more errors . the proposed method is based on two forms of cheating: providing a compressed representation of the target as an input and fine-tuning on the test set. |
Multi-Source Syntactic Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to integrate source syntax into neural machine translations use linearized parses. |
| Approach: | They propose a linearized parsed neural machine translation technique that integrates source syntax into neural machine learning. |
| Outcome: | The proposed model improves over seq2seq and parsed baselines by over 1 BLEU on the WMT17 English-German task. |