Giving Attention to the Unexpected: Using Prosody Innovations in Disfluency Detection (N19-1)
Copied to clipboard
| Challenge: | Disfluencies in spontaneous speech are associated with prosodic disruptions. |
| Approach: | They propose a method to extract acoustic-prosodic cues from word transcripts . they explore early and late fusion techniques for integrating text and prosody . |
| Outcome: | The proposed approach shows gains over a high-accuracy text-only model. |
Similar Papers
Integrating Disfluency-based and Prosodic Features with Acoustics in Automatic Fluency Evaluation of Spontaneous Speech (2020.lrec-1)
Copied to clipboard
| Challenge: | acoustics, prosody, and disfluency-based features are used to evaluate fluent/disfluent speech . filling pauses and word fragments are used for automatic fluency evaluation . |
| Approach: | They integrate acoustics, prosody, and disfluency-based features into an automatic fluency evaluation task. |
| Outcome: | The proposed model improves when integrated with prosodic features, but not when disfluent speech is detected. |
Parsing Speech: a Neural Approach to Integrating Lexical and Acoustic-Prosodic Information (N18-1)
Copied to clipboard
| Challenge: | acoustic signals provide cues that help listeners disambiguate difficult parses . speech carries useful extra information associated with prosodic structure . |
| Approach: | They propose a model that integrates transcribed text and acoustic-prosodic features into a neural network that accepts text and prosodic feature. |
| Outcome: | The proposed model improves parse and disfluency detection scores over a strong text-only baseline. |
Disfluent Cues for Enhanced Speech Understanding in Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a large number of language models struggle to handle disfluencies, authors say . when a speaker hesitates, interrupts themselves, repeats or corrects words, or abandons phrases, it can make their speech fragmented. |
| Approach: | They propose to use disfluent queries to “clean” spontaneous speech . they propose to apply disfluencies to models that use different types of speech repairs . |
| Outcome: | The proposed model improves on a reading comprehension task using disfluent queries . the results suggest that disfluencies can improve model performance, rather than their removal . |
Quantifying the redundancy between prosody and text (2023.emnlp-main)
Copied to clipboard
Lukas Wolf, Tiago Pimentel, Evelina Fedorenko, Ryan Cotterell, Alex Warstadt, Ethan Wilcox, Tamar Regev
| Challenge: | Existing studies suggest partial redundancy between prosody and linguistic information. |
| Approach: | They use large language models to estimate how much information is redundant between prosody and the words themselves. |
| Outcome: | The proposed model can predict prosodic features across prosodic features, including intensity, duration, pauses, and pitch contours. |
Prosody: Models, Methods, and Applications (2021.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial will overview the computational modeling of prosody. |
| Approach: | This tutorial will overview the computational modeling of prosody . it will discuss the latest advances in prosody and diverse applications . |
| Outcome: | This tutorial will overview the computational modeling of prosody. |
Harnessing Whisper for Prosodic Stress Analysis (2025.findings-acl)
Copied to clipboard
| Challenge: | Prosody affects how people produce and understand language, yet studies of how it does so have been hindered by the lack of efficient tools for analyzing prosodic stress. |
| Approach: | They propose to use OpenAI Whisper large-v2 to analyze prosodic stress . they propose to characterize decision boundaries used by Whisper for prosodic stress interpretation . |
| Outcome: | The proposed model can learn gender-specific stress patterns and transfer learning from one type of stress to another, surpassing traditional machine learning models. |
CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-Training (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for expressive text-to-speech only implicitly learn prosody with masked token reconstruction tasks. |
| Approach: | They propose a cross-modal contrastive pre-training framework that learns from prosody variance of the same text token under different contexts. |
| Outcome: | The proposed framework can learn from prosody variance of a text token under different contexts. |
DiffStyleTTS: Diffusion-based Hierarchical Prosody Modeling for Text-to-Speech with Diverse and Controllable Styles (2025.coling-main)
Copied to clipboard
| Challenge: | Existing models for text-to-speech (TTS) synthesize speech with acoustic features . autoregressive models have problems with word skipping and repeated reading . non-autoregressive acustic models lack probabilistic modeling and unimodal characteristics of Gaussian distribution don't conform to true distribution of aural features, which restricts the diversity of generated prosodic features. |
| Approach: | They propose a multi-speaker acoustic model that hierarchically models speech prosodic features and controls different prosodic styles to guide prosody prediction. |
| Outcome: | The proposed method outperforms baseline models in naturalness and achieves superior synthesis speed compared to baseline models. |
End-to-End Speech Recognition and Disfluency Removal (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Disfluency detection is usually an intermediate step between an automatic speech recognition system and a downstream task. |
| Approach: | They propose to train models to directly map disfluent speech into fluent transcripts without relying on a separate disfluency detection model. |
| Outcome: | The proposed models learn to generate fluent transcripts, but their performance is slightly worse than a baseline pipeline approach consisting of an ASR system and a specialized disfluency detection model. |
Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing direct S2TT systems have been unable to disambiguate utterances where prosody plays a crucial role. |
| Approach: | They propose to use contrastive evaluation to quantitatively measure the ability of direct S2TT systems to disambiguate utterances where prosody plays a crucial role. |
| Outcome: | The proposed system improves overall accuracy 12.9% and improves intent scores 15.6%. |