Papers by Felix Stahlberg
Multi-representation ensembles and delayed SGD updates improve syntax-based NMT (P18-2)
Copied to clipboard
| Challenge: | Existing studies have shown that NMT models trained to generate target syntax exhibit improved sentence structure relative to those trained on plain-text. |
| Approach: | They propose an approach to decoding ensembles of models generating different representations, focusing on models generating syntax. |
| Outcome: | The proposed approach gives state-of-the-art performance on a difficult Japanese-English task. |
The Role of Outgoing Connection Heterogeneity in Feedforward Layers of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using a new fine-tuning loss, we show that inner neurons with diverse outgoing connections are more critical to model performance than those with uniform connections. |
| Approach: | They propose a new loss that reduces the outgoing connection entropy in feedforward layers and elucidates the role of outgoing connections in large language models. |
| Outcome: | The proposed method is significantly more effective than removing neurons randomly or based on their magnitude. |
Using Context in Neural Machine Translation Training Objectives (2020.acl-main)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) training is based on document-level metrics, not sentence-level BLEU. |
| Approach: | They propose to merge document-level metrics with batch-level documents to improve NMT training. |
| Outcome: | The proposed training is more robust for document-level metrics than sequence MRT and maximum-likelihood training. |
Neural Grammatical Error Correction with Finite State Transducers (N19-1)
Copied to clipboard
| Challenge: | Language model based GEC (LM-GEC) is a promising alternative to SMT and neural sequence-to-sequence models. |
| Approach: | They propose to use finite state transducers to improve LM-GEC by rescoring with neural language models. |
| Outcome: | The proposed model outperforms the best published results on the CoNLL-2014 test set and achieves far better relative improvements over the baselines. |
Jam or Cream First? Modeling Ambiguity in Neural Machine Translation with SCONES (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing neural machine translation models learn the probability P (y|x) of the target sentence given the source sentence x. |
| Approach: | They propose to replace softmax activation with a multi-label classification layer that can model ambiguity more effectively. |
| Outcome: | The proposed multi-label classification layer can model ambiguity more effectively . it yields consistent BLEU score gains across six translation directions . |
Towards an On-device Agent for Text Rewriting (2024.findings-naacl)
Copied to clipboard
Yun Zhu, Yinxiao Liu, Felix Stahlberg, Shankar Kumar, Yu-Hui Chen, Liangchen Luo, Lei Shu, Renjie Liu, Jindong Chen, Lei Meng
| Challenge: | Large Language Models (LLMs) have demonstrated impressive capabilities for text rewriting, however creating a smaller yet potent language model presents two formidable challenges: costly data collection and absence of emergent capabilities. |
| Approach: | They propose a new instruction tuning method to develop a mo-bile text rewriting model that leverages LLM-generated data and heuristic reinforcement learning, eliminating the need for human data collection. |
| Outcome: | The proposed model surpasses the current state-of-the-art LLMs in text rewriting while maintaining a significantly reduced model size using public benchmark EditEval and our new benchmark. |
Uncertainty Determines the Adequacy of the Mode and the Tractability of Decoding in Sequence-to-Sequence Models (2022.acl-long)
Copied to clipboard
| Challenge: | In many natural language processing tasks the same input can have multiple possible outputs. |
| Approach: | They propose a novel exact n-best search algorithm for neural sequence models to measure sentence-level uncertainty by computing the degree of overlap between references from two different NLP tasks. |
| Outcome: | The proposed algorithm overly spreads the probability mass for uncertain tasks and sentences. |
Text Generation with Text-Editing Models (2022.naacl-tutorials)
Copied to clipboard
Eric Malmi, Yue Dong, Jonathan Mallinson, Aleksandr Chuklin, Jakub Adamek, Daniil Mirylenka, Felix Stahlberg, Sebastian Krause, Shankar Kumar, Aliaksei Severyn
| Challenge: | Text-editing models are a popular alternative to seq2seq for monolingual text generation tasks such as text summarization and style transfer. |
| Approach: | They propose to use text-editing models to predict edit operations applied to the source sequence and to generate outputs word-by-word from scratch. |
| Outcome: | This paper provides an overview of the text-edit based models and their current state-of-the-art approaches. |
Domain Adaptive Inference for Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Neural Machine Translation models are effective when trained on broad domains with large datasets, such as news translation. |
| Approach: | They propose a novel approach for adaptive ensemble weighting for Neural Machine Translation by extending Bayesian Interpolation with source information. |
| Outcome: | The proposed approach improves performance on Spanish-English and English-German tasks without the need for the domain label. |
Seq2Edits: Sequence Transduction Using Span-level Edit Operations (2020.emnlp-main)
Copied to clipboard
| Challenge: | Seq2Edits is an open-vocabulary approach to sequence editing for natural language processing tasks with a high degree of overlap between input and output texts. |
| Approach: | They propose an open-vocabulary approach to sequence editing for NLP tasks with a high degree of overlap between input and output texts. |
| Outcome: | The proposed approach speeds up inference by up to 5.2x compared to full sequence models . it improves explainability by associating each edit operation with a human-readable tag. |
Long-Form Speech Translation through Segmentation with Finite-State Decoding Constraints on Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a challenge in speech translation is that plenty of spoken content is long-form, but short units are necessary for obtaining high-quality translations. |
| Approach: | They propose a large language model to split long ASR transcripts into segments that can be independently translated to maximize translation quality. |
| Outcome: | The proposed model improves the average BLEU by 2.9 points for English–German, English–Spanish, and English–Arabic TED talk translation in 9 sets. |
On NMT Search Errors and Model Errors: Cat Got Your Tongue? (D19-1)
Copied to clipboard
| Challenge: | We show that at the root of the problem of empty translations lies an inherent bias towards shorter translations. |
| Approach: | They propose an exact inference procedure for neural sequence models based on beam search and depth-first search. |
| Outcome: | The proposed procedure finds that beam search fails to find the best model scores . the results show that the model often prefers an empty translation . |