Papers by Kevin Duh

Handshape-Aware Sign Language Recognition: Extended Datasets and Exploration of Handshape-Inclusive Methods (2023.findings-emnlp)

Challenge:	evaluating machine translation (MT) with cross-lingual information retrieval is relatively time-consuming and subjective.
Approach:	They propose a toolkit that evaluates machine translation with a proxy task of cross-lingual information retrieval.
Outcome:	The proposed toolkit is based on the "metrics shared task" of WMT2019.

Copied to clipboard

Xuan Zhang, Kevin Duh

Offer a Different Perspective: Modeling the Belief Alignment of Arguments in Multi-party Debates (2022.emnlp-main)

Challenge:	Existing work on sign language recognition encodes videos without acknowledging phonological attributes of signs.
Approach:	They propose a single-encoder network and a dual-encoding network for handshape-inclusive sign language recognition.
Outcome:	The proposed methods outperform baseline methods in the PHOENIX14T-HS dataset . the proposed methods consistently outperformed baseline methods .

Copied to clipboard

Suzanna Sia, Kokil Jaidka, Hansin Ahuja, Niyati Chhaya, Kevin Duh

Reasoning

Challenge:	Existing work on persuasion in online forums focuses on identifying debate winners and winning negotiation games.
Approach:	They adopt a hierarchical generative Variational Autoencoder model to model winning arguments . they propose competing hypotheses about the nature of argumentation .
Outcome:	The proposed model predicts winning arguments in reddit debates . it uses a hierarchical generative Variational Autoencoder to model argumentation .

ESPnet-ST: All-in-One Speech Translation Toolkit (2020.acl-demos)

Copied to clipboard

Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, Shinji Watanabe

Reproducible and Efficient Benchmarks for Hyperparameter Optimization of Neural Machine Translation Systems (2020.tacl-1)

Challenge:	ESPnet-ST is a new project for the quick development of speech-to-speech translation systems.
Approach:	They propose a framework for rapid development of speech-to-speech translation systems . they provide all-in-one recipes including data pre-processing, feature extraction, training, and decoding pipelines .
Outcome:	The proposed model outperforms the current state-of-the-art models on a wide range of benchmark datasets.

Copied to clipboard

Xuan Zhang, Kevin Duh

Responsible & Trustworthy NLP

Challenge:	Optimal versus suboptimal hyperparameters can lead to dramatic swings in system performance.
Approach:	They propose to use a library of pre-trained models for fast, low cost HPO experimentation and to propose metrics for evaluating HPO methods on NMT.
Outcome:	The proposed method uses a library of pre-trained models for fast, low cost experimentation on neural machine translation (NMT) .

The Effects of Language Token Prefixing for Multilingual Machine Translation (2022.aacl-short)

Copied to clipboard

Rachel Wicks, Kevin Duh

Leveraging End-to-End ASR for Endangered Language Documentation: An Empirical Study on Yolóxochitl Mixtec (2021.eacl-main)

Challenge:	In recent years, the field has moved towards large neural models either translating from or into many languages.
Approach:	They propose to prefix language tokens onto a source or target sequence to improve translation performance.
Outcome:	The proposed methods improve translation performance and source side prefixes improve translation.

Copied to clipboard

Jiatong Shi, Jonathan D. Amith, Rey Castillo García, Esteban Guadalupe Sierra, Kevin Duh, Shinji Watanabe

Sequence Models for Computational Etymology of Borrowings (2021.findings-acl)

Challenge:	End-to-end ASR systems that eschew linguistic resources but are more dependent on large-data settings are suggested as a solution to EL documentation bottlenecks.
Approach:	They propose to build an end-to-end ASR system that is reproducible by the ASR community and propose a novice transcription correction task.
Outcome:	The proposed method would mitigate bottlenecks and shortages in transcribers . it is based on a Yoloxóchitl Mixtec corpus and is reproducible by the ASR community.

Copied to clipboard

Winston Wu, Kevin Duh, David Yarowsky

Stochastic Answer Networks for Machine Reading Comprehension (P18-1)

Challenge:	a computational model of word borrowing can be useful for lexicon expansion and language preservation.
Approach:	They propose to use neural sequence models to model word borrowings from a donor word to an incorporated word.
Outcome:	The proposed model beats baseline models in both directions, with the quantity of data strongly influencing performance.

Copied to clipboard

Xiaodong Liu, Yelong Shen, Kevin Duh, Jianfeng Gao

Reasoning

Challenge:	Several recent MRC models employ multi-step reasoning . we show that the use of a stochastic prediction dropout improves robustness .
Approach:	They propose a stochastic answer network that simulates multi-step reasoning in machine reading comprehension.
Outcome:	The proposed model improves robustness and results competitive with state-of-the-art models on the Stanford Question Answering Dataset and Microsoft MAchine Reading COmprehension Dataset.

AfriCLIRMatrix: Enabling Cross-Lingual Information Retrieval for African Languages (2022.emnlp-main)

Copied to clipboard

Odunayo Ogundepo, Xinyu Zhang, Shuo Sun, Kevin Duh, Jimmy Lin

Adaptive Mixed Component LDA for Low Resource Topic Modeling (2021.eacl-main)

Challenge:	Existing datasets for cross-lingual information retrieval are limited in many languages, especially those spoken in Africa.
Approach:	They propose to build a test collection for cross-lingual information retrieval in 15 diverse African languages.
Outcome:	AfriCLIRMatrix contains 6 million queries in English and 23 million relevance judgments automatically mined from Wikipedia inter-language links, covering many more African languages than any existing information retrieval test collection.

Copied to clipboard

Suzanna Sia, Kevin Duh

Responsible & Trustworthy NLP

Challenge:	Probabilistic topic models in low resource settings are faced with less reliable estimates due to sparsity of discrete word co-occurrence counts.
Approach:	They propose a mixture model which interpolates between discrete and continuous topic-word distributions and utilises pre-trained embeddings to improve topic coherence.
Outcome:	The proposed model outperforms fully discrete, fully continuous, and static mixture models on topic coherence in low resource settings.

HABLex: Human Annotated Bilingual Lexicons for Experiments in Machine Translation (D19-1)

Copied to clipboard

Brian Thompson, Rebecca Knowles, Xuan Zhang, Huda Khayrallah, Kevin Duh, Philipp Koehn

Bilingual Lexicon Induction for Low-Resource Languages using Graph Matching via Optimal Transport (2022.emnlp-main)

Challenge:	Existing methods to incorporate bilingual lexicons into statistical machine translation are unclear how to do so in the neural framework.
Approach:	They present a dataset to test methods for bilingual lexicon integration into neural machine translation using human generated alignments of words and phrases in three language pairs.
Outcome:	The proposed method improves on baselines and improves training to address overfitting.

Copied to clipboard

Kelly Marchisio, Ali Saad-Eldin, Kevin Duh, Carey Priebe, Philipp Koehn

Structured Data in NLP

Challenge:	Existing literature on bilingual lexicon induction fails in low-resource scenarios . a language dataset is considered low- resource based on its own embedding space .
Approach:	They propose a graph-matching method that improves bilingual lexicon induction performance across 40 language pairs using optimal transport.
Outcome:	The proposed method is especially strong with low amounts of supervision.

Broad-Coverage Semantic Parsing as Transduction (D19-1)

Copied to clipboard

Sheng Zhang, Xutai Ma, Kevin Duh, Benjamin Van Durme

Anti-LM Decoding for Zero-shot In-context Machine Translation (2024.findings-naacl)

Challenge:	Existing approaches to broad-coverage semantic parsing are not applicable to all frameworks because of the lack of explicit alignments between tokens in the sentence and nodes in the semantic graph.
Approach:	They propose a transduction parsing paradigm that unifies different broad-coverage semantic parsers into a paradigm that leverages multiple attention mechanisms to build meaning representation.
Outcome:	The proposed approach improves state-of-the-art on AMR, SDP and UCCA and is competitive with the state- of-the art on SDP.

Copied to clipboard

Suzanna Sia, Alexandra DeLucia, Kevin Duh

Query Decomposition for RAG: Balancing Exploration-Exploitation (2026.eacl-long)

Challenge:	Existing approaches to zero-shot learning with large language models are poorly calibrated for zero-shoot tasks.
Approach:	They propose a contrastive decoding objective with a decay factor to address in-context bias . they conduct experiments on 3 model types and sizes, 3 language directions, and beam search .
Outcome:	The proposed method outperforms state-of-the-art decoding objectives with 20 BLEU points improvement from the default objective in some settings.

Copied to clipboard

Roxana Petcu, Kenton Murray, Daniel Khashabi, Evangelos Kanoulas, Maarten de Rijke, Dawn Lawrie, Kevin Duh

Information Retrieval

Challenge:	Complex user queries often involve the exclusion of information, negation, or missing entities.
Approach:	They propose to decompose user requests into subqueries, retrieve potentially relevant documents for each and then aggregate them to generate an answer.
Outcome:	The proposed method achieves 35% gain in document-level precision and 15% increase in -nDCG . it also improves the downstream task of long-form generation.

AskQE: Question Answering as Automatic Evaluation for Machine Translation (2025.findings-acl)

Copied to clipboard

Dayeon Ki, Kevin Duh, Marine Carpuat

Text Generation

Challenge:	Existing MT error detection and quality estimation (QE) techniques do not address this practical scenario.
Approach:	They propose a question generation and answering framework that detects critical MT errors and provides actionable feedback to help users decide whether to accept or reject MT outputs even without the knowledge of the target language.
Outcome:	The proposed framework has higher Kendall’s Tau correlation and decision accuracy with human ratings compared to other QE metrics.

AutoML for NLP (2023.eacl-tutorials)

Copied to clipboard

Kevin Duh, Xuan Zhang

Challenge:	Automated Machine Learning (AutoML) is an emerging field that has potential to impact how we build models in NLP.
Approach:	This tutorial will summarize the main AutoML techniques and illustrate how to apply them to improve the NLP model-building process.
Outcome:	This tutorial summarizes the main AutoML techniques and illustrates how to apply them to improve the model-building process.

Exploring Geometric Representational Disparities between Multilingual and Bilingual Translation Models (2024.lrec-main)

Copied to clipboard

Neha Verma, Kenton Murray, Kevin Duh

Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation (N19-1)

Challenge:	Existing work shows that limited modeling capacity is a major contributor to reduced performance in multilingual models.
Approach:	They investigate the isotropy of multilingual model decoder representations using intrinsic dimensionality and IsoScore to measure how they utilize the dimensions in their underlying vector space.
Outcome:	The proposed model decoder representations are less isotropic and occupy fewer dimensions than bilingual models.

Copied to clipboard

Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, Philipp Koehn

Curriculum Learning for Domain Adaptation in Neural Machine Translation (N19-1)

Challenge:	Neural Machine Translation (NMT) performs poorly without large training corpora.
Approach:	They propose a machine learning method that retains the majority of general-domain performance lost in continued training without degrading in-domain.
Outcome:	The proposed method retains the majority of general-domain performance lost in continued training without degrading in-domain performances.

Copied to clipboard

Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, Kevin Duh

Cross-lingual Decompositional Semantic Parsing (D18-1)

Challenge:	Neural machine translation (NMT) performance drops when domains do not match and in-domain training data is scarce.
Approach:	They propose a curriculum learning approach to adapt generic neural machine translation models to a specific domain.
Outcome:	The proposed approach outperforms unadapted and adapted baselines in two domains and two language pairs.

Copied to clipboard

Sheng Zhang, Xutai Ma, Rachel Rudinger, Kevin Duh, Benjamin Van Durme

Data and Parameter Scaling Laws for Neural Machine Translation (2021.emnlp-main)

Challenge:	Renewed interest in semantic analysis has led to a surge of proposed new frameworks . many of these efforts are limited to the analysis of English, but with a number of exceptions e.g., recent efforts in Minimal Recursion Semantics (MRS) and multilingual FrameNet annotation and parsing.
Approach:	They propose a cross-lingual decompositional semantic analysis task based on a target language . they propose 'end-to-end' model with an annotating mechanism that supports intra-sentential coreference .
Outcome:	The proposed model outperforms baselines by at least 1.75 F1 score on an evaluation dataset.

Copied to clipboard

Mitchell A Gordon, Kevin Duh, Jared Kaplan

Benchmarking Neural and Statistical Machine Translation on Low-Resource African Languages (2020.lrec-1)

Challenge:	Recent work shows that supervised neural machine translation models scale like a power law with the amount of training data and number of non-embedding parameters in the model.
Approach:	They show that cross-entropy loss of supervised neural machine translation models scales like a power law with the amount of training data and number of non-embedding parameters in the model.
Outcome:	The proposed model can predict BLEU and ROI of labeling data in low-resource language pairs.

Copied to clipboard

Kevin Duh, Paul McNamee, Matt Post, Brian Thompson

Text Generation

Challenge:	a recent study has focused on languages where large amounts of resources are available.
Approach:	They benchmark state of the art statistical and neural machine translation systems on Somali and Swahili languages . they find that statistical machine translation and neural translation can perform similarly in low-resource scenarios .
Outcome:	The results show that statistical machine translation and neural machine translation perform similarly in low-resource scenarios.

SpeechQE: Estimating the Quality of Direct Speech Translation (2024.emnlp-main)

Copied to clipboard

HyoJung Han, Kevin Duh, Marine Carpuat

Speech & Audio in NLP

Challenge:	Recent advances in automatic quality estimation for machine translation focus on written language, leaving the speech modality underexplored.
Approach:	They propose a new quality estimation system based on cascaded and end-to-end architectures.
Outcome:	The proposed system is better suited to estimating the quality of direct speech translation than existing systems designed for text translation.

AMR Parsing as Sequence-to-Graph Transduction (P19-1)

Copied to clipboard

Sheng Zhang, Xutai Ma, Kevin Duh, Benjamin Van Durme

Should I Share this Translation? Evaluating Quality Feedback for User Reliance on Machine Translation (2025.emnlp-main)

Challenge:	Abstract Meaning Representation (AMR) parsing is the task of transducing natural language text into AMR, a graphbased formalism used for capturing sentence-level semantics.
Approach:	They propose a model that treats AMR parsing as sequence-to-graph transduction by aligner-free, and can be effectively trained with limited amounts of labeled AMR data.
Outcome:	The proposed model outperforms all previously reported SMATCH scores on AMR 2.0 (76.3%) and AMR 1.0 (70.2%).

Copied to clipboard

Dayeon Ki, Kevin Duh, Marine Carpuat

A Hyperparameter Optimization Toolkit for Neural Machine Translation Research (2023.acl-demo)

Challenge:	Existing studies on the impact of feedback on human decision-making are limited as people are not equipped to assess the quality of AI predictions.
Approach:	They compare the quality of MT inputs and outputs with explicit and implicit feedbacks that directly give users an assessment of translation quality using error highlights and LLM explanations.
Outcome:	The proposed model improves decision accuracy and appropriate reliance by using error highlights and explanations, and by using backtranslation and question–answer tables.

Copied to clipboard

Xuan Zhang, Kevin Duh, Paul McNamee

CLIRMatrix: A massively large collection of bilingual and multilingual datasets for Cross-Lingual Information Retrieval (2020.emnlp-main)

Challenge:	Insufficient tuning of hyperparameters may lead to poor results and exaggerated results . lack of open-source support tools means that the level of rigor in hyperparametric optimization may vary widely.
Approach:	They propose a hyperparameter optimization toolkit for neural machine translation that is implemented as a wrapper on top of the open-source Sockeye NMT software.
Outcome:	The proposed toolkit is implemented as a wrapper on top of the open-source Sockeye NMT software.

Copied to clipboard

Shuo Sun, Kevin Duh

IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces (2022.emnlp-main)

Challenge:	Cross-Lingual Information Retrieval (CLIR) is a retrieval task in which search queries and candidate documents are written in different languages.
Approach:	They present a massively large collection of bilingual and multilingual datasets for Cross-Lingual Information Retrieval extracted automatically from Wikipedia.
Outcome:	The proposed datasets are the largest and most comprehensive CLIR dataset to date.

Copied to clipboard

Kelly Marchisio, Neha Verma, Kevin Duh, Philipp Koehn

Representation Learning

Challenge:	a faulty cross-lingual mapping technique results in non-isomorphic word embedding spaces . we address the root-cause of this problem by increasing the relative isomorphism of word embedsing spaces.
Approach:	They address the root-cause of faulty cross-lingual mapping by incorporating global measures of isomorphism into the skipgram loss function.
Outcome:	The proposed method improves bilingual lexicon induction under domain mismatch and with training algorithm dissimilarities.

An Extensive Exploration of Back-Translation in 60 Languages (2023.findings-acl)

Copied to clipboard

Paul McNamee, Kevin Duh

Machine Translation

Challenge:	Back-translation has been shown to improve model quality through the creation of synthetic training bitext.
Approach:	They use back-translation to train models from 60 languages into English . early studies showed promise of the technique and follow on studies have produced refinements .
Outcome:	a new study shows that back-translation improves translation quality in low-resource languages . the results are consistent with previous studies, though there are limitations .

An Analysis of Euclidean vs. Graph-Based Framing for Bilingual Lexicon Induction from Word Embedding Spaces (2021.findings-emnlp)

Copied to clipboard

Kelly Marchisio, Youngser Park, Ali Saad-Eldin, Anton Alyakin, Kevin Duh, Carey Priebe, Philipp Koehn

Whisper-UT: A Unified Translation Framework for Speech and Text (2025.emnlp-main)

Challenge:	Existing work in bilingual lexicon induction views word embeddings as vectors in Euclidean space.
Approach:	They propose to use word embeddings as nodes in a weighted graph to examine a node’s graph neighborhood without assuming a linear transform.
Outcome:	The proposed approaches are compared under different data conditions and show that they complement each other when combined.

Copied to clipboard

Cihan Xiao, Matthew Wiesner, Debashish Chakraborty, Reno Kriz, Keith Cunningham, Kenton Murray, Kevin Duh, Luis Tavarez-Arce, Paul McNamee, Sanjeev Khudanpur

Large-Scale Bitext Corpora Provide New Evidence for Cognitive Representations of Spatial Terms (2024.eacl-long)

Challenge:	Encoder-decoder models have achieved remarkable success in speech and text tasks, but efficiently adapting them to diverse uni/multimodal scenarios remains a challenge.
Approach:	They propose a framework that leverages lightweight adapters to enable seamless adaptation across tasks.
Outcome:	The proposed framework improves speech translation performance through a 2-stage decoding strategy without requiring 3-way parallel data.

Copied to clipboard

Peter Viechnicki, Kevin Duh, Anthony Kostacos, Barbara Landau

Representation Learning

Challenge:	Recent evidence suggests that there exist two classes of cognitive representations within the spatial terms of a language.
Approach:	They propose a pipeline for extracting, isolating, and aligning spatial terms from parallel text . they find evidence that variability in functional terms differs significantly from that of geometric terms .
Outcome:	The proposed pipeline extracts, isolates, and aligns spatial terms in basic locative constructions from parallel text.

An Interdisciplinary Approach to Human-Centered Machine Translation (2025.emnlp-main)

Copied to clipboard

Marine Carpuat, Omri Asscher, Kalika Bali, Luisa Bentivogli, Fred Blain, Lynne Bowker, Monojit Choudhury, Hal Daumé Iii, Kevin Duh, Ge Gao, Alvin C Grissom II, Marzena Karpinska, Elaine C Khoong, William D. Lewis, Andre Martins, Mary Nurminen, Douglas W. Oard, Maja Popovic, Michel Simard, François Yvon