Papers by David Harwath
Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent visuolinguistic pre-trained models fail miserably on the Winoground dataset, which challenges models to match paired images and English captions. |
| Approach: | They propose to annotate a Winoground dataset that challenges visuolinguistic models to match paired images and English captions with items constructed to overlap lexically but differ in meaning. |
| Outcome: | The proposed dataset challenges models to match paired images and English captions with items constructed to overlap lexically but differ in meaning. |
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models for synthesising fluent, natural-sounding spoken audio captions do not require natural language text as an intermediate representation or source of supervision. |
| Approach: | They propose a model for directly synthesizing fluent, natural-sounding spoken audio captions for images that does not require natural language text as an intermediate representation or source of supervision. |
| Outcome: | The proposed model captures diverse visual semantics of images and can replace text with a set of discrete, sub-word speech units. |
Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration (2026.findings-acl)
Copied to clipboard
Ryan Soh-Eun Shim, Kwanghee Choi, Kalvin Chang, Ming-Hao Hsu, Florian Eichin, Zhizheng Wu, Alane Suhr, Michael A. Hedderich, David Harwath, David R. Mortensen, Barbara Plank
| Challenge: | We show that script information is linearly encoded in the activation space of multilingual speech models . modifying activations at inference time induces script change even in unconventional pairings . |
| Approach: | They propose to add script vectors to activations at test time to induce script change . they also show that script information is linearly encoded in the activation space of multilingual speech models . |
| Outcome: | The proposed approach can induce script change even in unconventional language-script pairings. |
VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild (2024.acl-long)
Copied to clipboard
| Challenge: | VoiceCraft is a token-infilling neural codec language model for speech editing and zero-shot text-to-speech evaluation. |
| Approach: | They introduce a token infilling neural codec language model that performs on speech editing and zero-shot text-to-speech tasks. |
| Outcome: | The proposed model outperforms previous models on speech editing and zero-shot text-to-speech tasks. |
[b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on how self-supervised speech models encode rich phonetic information have not explored how they are structured. |
| Approach: | They conduct a comprehensive analysis of the underlying structure of S3M representations with particular attention to phonological vectors. |
| Outcome: | The proposed model encodes phonologically interpretable and compositional vectors, demonstrating phonology vector arithmetic. |
Speak: A Toolkit Using Amazon Mechanical Turk to Collect and Validate Speech Audio Recordings (2022.lrec-1)
Copied to clipboard
| Challenge: | Speak is a toolkit that allows researchers to crowdsource speech recordings using Amazon Mechanical Turk (MTurk). |
| Approach: | They propose to use Amazon Mechanical Turk to crowdsource speech recordings . they use various measures to ensure that the recordings are of adequate quality . |
| Outcome: | Speak is an open-source toolkit that allows researchers to crowdsource speech recordings using Amazon Mechanical Turk (MTurk). |
Textless Speech-to-Speech Translation With Limited Parallel Data (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing speech-to-speech translation models either leverage text as an intermediate step or require hundreds of hours of parallel speech data. |
| Approach: | They propose a framework for training textless S2ST models that require dozens of hours of parallel speech data. |
| Outcome: | The proposed model achieves reasonable performance on three domains with single-speaker synthesized speech. |
VoiceStar: Robust Zero-Shot Autoregressive TTS with Duration Control and Extrapolation (2026.findings-acl)
Copied to clipboard
| Challenge: | Neural codec language models (NCLMs) lack fine-grained controllability and inability to extrapolate to sequence lengths much longer than those seen during training. |
| Approach: | They propose a novel autoregressive encoder-decoder neural codec language model that can be trained with a Continuation-Prompt Mixed training system. |
| Outcome: | The proposed model outperforms or is on par with current state-of-the-art models on short-form benchmarks such as LibriSpeech and Seed-TTS in terms of intelligibility and naturalness. |
Multimodal Contextualized Semantic Parsing from Speech (2024.acl-long)
Copied to clipboard
| Challenge: | Towards this goal, we introduce Semantic Parsing in Contextual Environments (SPICE) task designed to enhance artificial agents’ contextual awareness by integrating multimodal inputs with prior contexts. |
| Approach: | They introduce a task designed to enhance artificial agents’ contextual awareness by integrating multimodal inputs with prior contexts. |
| Outcome: | The proposed task is based on the VG-SPICE dataset and the Audio-Vision Dialogue Scene Parser (AViD-SP) it allows agents to maintain their contextual state within a structured, dense information framework that is scalable and interpretable . |
Scaling Rich Style-Prompted Text-to-Speech Datasets (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets that only cover basic tags are limited in their scale or coverage of style tags. |
| Approach: | They propose a large-scale dataset that annotates speech utterances with rich style captions. |
| Outcome: | The proposed dataset scales speech utterances with rich style captions for the first time. |
When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants (2023.acl-short)
Copied to clipboard
| Challenge: | Existing models focus on improving the efficiency of self-attention, but in practice they may be slower, especially given modest input lengths that are typical of many tasks. |
| Approach: | They propose a novel local-attention variant of a self-supervised speech model that uses input length thresholds to identify bottlenecks. |
| Outcome: | The proposed model is based on a self-attention-based model with a high input length threshold. |
VoiceCraft-X: Unifying Multilingual, Voice-Cloning Speech Synthesis and Speech Editing (2025.emnlp-main)
Copied to clipboard
Zhisheng Zheng, Puyuan Peng, Anuj Diwan, Cong Phuoc Huynh, Xiaohang Sun, Zhu Liu, Vimal Bhat, David Harwath
| Challenge: | Autoregressive language model for multilingual speech editing and zero-shot text-to-speech synthesis is available in 11 languages. |
| Approach: | They introduce an autoregressive neural codec language model which unifies multilingual speech editing and zero-shot text-to-speech synthesis across 11 languages. |
| Outcome: | The model generates high-quality, natural-sounding speech, even with limited per-language data . it shows robust performance in diverse linguistic settings, even in limited per language data compared to other models . |