Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions (2026.acl-long)
Copied to clipboard
| Challenge: | Recent Spoken Language Models lack the capability to discern Third-Party Interruptions (TPI) from the primary user’s ongoing flow, leaving them vulnerable to contextual failures. |
| Approach: | They propose a dataset with speaker-aware hard negatives to enforce acoustic cue prioritization for interruption handling and a framework to measure the interruption-handling strategy and precise speaker discrimination in deceptive contexts. |
| Outcome: | The proposed framework mitigates semantic shortcut learning while neglecting acoustic signals essential for discerning speaker changes. |
Similar Papers
PRoDeliberation: Parallel Robust Deliberation for End-to-End Spoken Language Understanding (2024.findings-emnlp)
Copied to clipboard
Trang Le, Daniel Lazar, Suyoun Kim, Shan Jiang, Duc Le, Adithya Sagar, Aleksandr Livshits, Ahmed Aly, Akshat Shrivastava
| Challenge: | End-to-end models for Spoken Language Understanding have been autoregressive, resulting in higher latencies. |
| Approach: | They propose a method that uses Connectionist Temporal Classification to train robust non-autoregressive deliberation models. |
| Outcome: | The proposed method achieves 10x latency reduction over autoregressive models while preserving ability to correct ASR mistranscriptions. |
It’s Not under the Lamppost: Expanding the Reach of Conversational AI (2024.lrec-main)
Copied to clipboard
| Challenge: | Focused probes into the capabilities of language-based assistants easily reveal significant areas of brittleness that demonstrate large gaps in their coverage. |
| Approach: | They propose a process for collecting specific kinds of data to uncover these gaps and an annotation scheme for system responses. |
| Outcome: | The proposed system includes both Conventional and GenAI systems, including ChatGPT and Bard/Gemini. |
Mind the Pause: Disfluency-Aware Objective Tuning for Multilingual Speech Correction with LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Spontaneous speech is rarely fluent, and disfluencies can degrade readability and reliability . a sequence tagger first marks disfluent tokens, and these signals guide instruction fine-tuning . |
| Approach: | They propose a multilingual correction pipeline where a sequence tagger first marks disfluent tokens . they add a contrastive learning objective that penalizes the reproduction of disfluency tokens. |
| Outcome: | The proposed model improves readability and reliability of ASR transcripts in three languages . disfluencies can cause misinterpretations, incoherent responses, poor user experience . |
Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representations (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study evaluated the extent to which SLMs encode nuanced syntactic and conceptual features . acoustic and phonetic features are shallow, but the extent of nuance is unclear . |
| Approach: | a new study evaluates contextual syntactic and semantic features in transformer-based speech language models . authors compare SLMs to linguistic competence assessments for large language models. |
| Outcome: | a new study compares SLMs with linguistic competence assessments to assess speech recognition and understanding . the results show that SLM models encode grammatical features more robustly than conceptual ones . |
VoiceBench: Benchmarking LLM-Based Voice Assistants (2026.tacl-1)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have enabled real-time speech interactions through LLMs. |
| Approach: | They propose a benchmark specifically designed to assess LLM-based voice assistants. |
| Outcome: | The proposed benchmark measures the performance of LLM-based voice assistants across eight tasks. |
DialDefer: A Framework for Detecting and Mitigating LLM Dialogic Deference (2026.acl-long)
Copied to clipboard
Parisa Rabbani, Priyam Sahoo, Ruben Mathew, Aishee Mondal, Harshita Ketharaman, Nimet Beyza Bozdag, Dilek Hakkani-Tür
| Challenge: | a single model can shift toward disagreement (skepticism) on graduate-level science and toward agreement (deference) on social judgment. |
| Approach: | They propose a framework to detect and mitigat framing-induced judgment shifts . they propose 'DialDefer' framework to help model disagreements and disagreements based on attribution . |
| Outcome: | The proposed framework detects and mitigates dialogic deference shifts in LLMs . human-vs-LLM attribution drives the largest shifts (17.7 pp swing) |
Evaluating Robustness of Large Audio Language Models to Audio Injection: An Empirical Study (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Audio-Language Models (LALMs) are increasingly being deployed in real-world applications, yet their robustness against malicious audio injection remains underexplored. |
| Approach: | They quantitatively assess their vulnerabilities and resilience using metrics: the Defense Success Rate, Context Robustness Score, and Judgment Robustic Index. |
| Outcome: | The proposed models demonstrate significant performance disparities across four attack scenarios. |
I Know, but I Don’t Know! How Persona Conflict Undermines Instruction Adherence in Large Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on persona-grounded dialogue assume idealized scenarios where persona and user utterances are fully aligned. |
| Approach: | They propose a taxonomy that categorizes model behaviors into three response types . they propose sycophantic, adherent, and wavering responses as response types. |
| Outcome: | The proposed framework categorizes model behaviors into three response types and develops a measurement schema grounded in this taxonomy. |
FalAI: A Dataset for End-to-end Spoken Language Understanding in a Low-Resource Scenario (2024.lrec-main)
Copied to clipboard
Andres Pineiro-Martin, Carmen Garcia-Mateo, Laura Docio-Fernandez, Maria del Carmen Lopez-Perez, Jose Gandarela-Rodriguez
| Challenge: | End-to-end (E2E) Spoken Language Understanding systems extract structured information from speech signals using a single model. |
| Approach: | They propose to use a dataset to extract structured information from speech signals . they define splits for noisy audio, hesitant audio and audio where sentence has changed . |
| Outcome: | The proposed model exploits acoustic information and avoids cascading errors . falAI dataset is the largest public SLU dataset in Galician and first to be obtained in low-resource scenario. |
Beyond Single-User Dialogue: Assessing Multi-User Dialogue State Tracking Capabilities of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have demonstrated remarkable performance in zero-shot dialogue state tracking (DST), reducing the need for task-specific training. |
| Approach: | They extend existing DST dataset by generating utterances of a second user based on speech act theory. |
| Outcome: | The proposed model incorporates utterances of a second user into conversations, enabling a controlled evaluation of LLMs in multi-user settings. |