Papers by Shyamnath Gollakota
Reading Between the Lines: The One-Sided Conversation Problem (2026.findings-acl)
Copied to clipboard
| Challenge: | In many real-world scenarios, only one side of a conversation is available for processing. |
| Approach: | They propose a one-sided conversation problem to reconstruct the missing speaker's turns and generate faithful summaries from one-side transcripts. |
| Outcome: | The proposed model improves reconstructions with prompting, but smaller models require fine tuning. |
LlamaPIE: Proactive In-Ear Conversation Assistants (2025.findings-acl)
Copied to clipboard
| Challenge: | LlamaPIE is the first real-time proactive assistant designed to enhance human conversations . it provides discreet, concise guidance delivered via hearable devices . traditional language models require explicit user invocation, but the assistant operates in the background . |
| Approach: | They propose a two-model pipeline that decides when to respond and a larger model that generates the response. |
| Outcome: | The proposed approach is effective in providing helpful, unobtrusive assistance on real-world datasets. |
AV-Dialog: Spoken Dialogue Models with Audio-Visual Input (2026.acl-long)
Copied to clipboard
| Challenge: | AV-Dialog uses audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. |
| Approach: | They propose a multimodal dialog framework that uses both audio and visual cues to track the target speaker. |
| Outcome: | AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction and human-rated dialogue quality. |
Proactive Hearing Assistants that Isolate Egocentric Conversations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing hearing assistants are "reactive" in that users manually prompt them to pick specific sound sources via spatial filtering or phone-based interfaces. |
| Approach: | They propose a dual-model architecture that uses the wearer's self-speech as an anchor to infer conversational partners and suppress others. |
| Outcome: | The proposed system can identify and separate conversation partners in multi-conversation settings without explicit user commands or prompts. |
Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing spoken dialogue models are half-duplex in nature and require explicit prompting by the user or implicit tracking of interruption or silence events. |
| Approach: | They propose to integrate time information into Llama3-8b so that they run synchronously with the real-world clock. |
| Outcome: | The proposed model outperforms state-of-the-art in dialogue meaningfulness while maintaining naturalness. |