EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wild (2025.findings-naacl)
Copied to clipboard
Junhyeok Kim, Min Soo Kim, Jiwan Chung, Jungbin Cho, Jisoo Kim, Sungwoong Kim, Gyeongbo Sim, Youngjae Yu
| Challenge: | EgoSpeak predicts when an agent should begin speaking based on egocentric streaming video. |
| Approach: | They propose a framework for real-time speech initiation prediction in egocentric streaming video by modeling the conversation from the camera wearer's first-person perspective. |
| Outcome: | The proposed framework outperforms random and silence-based baselines in real time and highlights the importance of multimodal input and context length in effectively deciding when to speak. |
Similar Papers
Proactive Hearing Assistants that Isolate Egocentric Conversations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing hearing assistants are "reactive" in that users manually prompt them to pick specific sound sources via spatial filtering or phone-based interfaces. |
| Approach: | They propose a dual-model architecture that uses the wearer's self-speech as an anchor to infer conversational partners and suppress others. |
| Outcome: | The proposed system can identify and separate conversation partners in multi-conversation settings without explicit user commands or prompts. |
Proactive Assistant Dialogue Generation from Streaming Egocentric Videos (2025.emnlp-main)
Copied to clipboard
Yichi Zhang, Xin Luna Dong, Zhaojiang Lin, Andrea Madotto, Anuj Kumar, Babak Damavandi, Joyce Chai, Seungwhan Moon
| Challenge: | Recent advances in conversational AI have been substantial, but developing real-time tasks guidance systems remains a challenge. |
| Approach: | They propose a data curation pipeline that synthesizes dialogues from annotated egocentric videos and a suite of automatic evaluation metrics that validated through extensive human studies. |
| Outcome: | The proposed framework synthesizes dialogues from annotated egocentric videos and validates them through extensive human studies. |
VoxMind: An End-to-End Agentic Spoken Dialogue System (2026.acl-long)
Copied to clipboard
Tianle Liang, Yifu Chen, Shengpeng Ji, Yijun Chen, Zhiyang Jia, Jingyu Lu, Fan Zhuo, Xueyi Pu, Yangzhuo Li, Zhou Zhao
| Challenge: | Existing research on end-to-end spoken dialogue models has focused on core perception and generation, with limited exploration of tool-augmented extensions. |
| Approach: | They propose a framework to equip end-to-end spoken dialogue models with comprehensive agentic abilities by leveraging a 470-hour AgentChat dataset. |
| Outcome: | The proposed framework outperforms Gemini-2.5-Pro on spoken agent tasks while maintaining general conversational quality. |
Spoken Conversational Agents with Large Language Models (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial focuses on the evolution of voice-native LLMs . it reviews the adaptation of text LLM to audio, cross-modal alignment, and joint speech–text training . |
| Approach: | This tutorial examines the evolution of voice-native LLMs in conversational agents . it compares cascaded and voice-based LLM systems to end-to-end retrieval-and vision-grounded systems . |
| Outcome: | This tutorial examines the evolution of voice-native LLMs . it compares the performance of voice assistants to current open-domain agents . |
A Dynamic Speaker Model for Conversational Interactions (N19-1)
Copied to clipboard
| Challenge: | a neural model for characterizing individual differences in speakers is shown to be useful in human-computer interaction and dialog act prediction. |
| Approach: | They propose a neural model for learning a dynamically updated speaker embedding in a conversational context. |
| Outcome: | The proposed model is used for content ranking and dialog act prediction in human-human conversations. |
Conversation Initiation by Diverse News Contents Introduction (N19-1)
Copied to clipboard
| Challenge: | Existing conversation systems assume that the user always initiates conversation and focus on how to respond to the given user’s utterance. |
| Approach: | They propose to generate initial utterance by summarizing and chatting about news articles to avoid boredom by relying on boilerplate utterrances like greetings. |
| Outcome: | The proposed model outperforms baseline models and based on information retrieval based and generation based models. |
Lexical Entrainment for Conversational Systems (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Conversational agents are expected to possess human-like features such as lexical entrainment (LE). |
| Approach: | They propose a dataset and a measure for LE for conversational systems to explicitly integrate LE into conversational system. |
| Outcome: | The proposed dataset and a measure for LE for conversational systems address this human-like phenomenon. |
Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for improving consistency often train with additional NLI labels or attach trained extra modules to the generative agent. |
| Approach: | They propose to encode personas into dialogue embeddings and a persona-conditioned dialogue dataset to improve persona consistency. |
| Outcome: | The proposed approach can enforce dialogue agents to refrain from contradictions and improve consistency of existing models. |
Mixed-Session Conversation with Egocentric Memory (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent dialogue systems exhibit an inability to replicate dynamic, continuous, long-term interactions involving multiple partners. |
| Approach: | They propose a multi-session dialogue system that builds on real-world interactions by integrating deep layered interactions and widening conversation networks. |
| Outcome: | The proposed system is based on a dataset of 6 consecutive dialogue episodes with four speakers (one main speaker and three partners) appearing in each episode. |
Game-Based Video-Context Dialogue (D18-1)
Copied to clipboard
| Challenge: | Current dialogue systems focus more on textual and speech context knowledge and are usually based on two speakers. |
| Approach: | They propose to use live soccer game videos and Twitch.tv chats to develop visual-grounded dialogue models. |
| Outcome: | The proposed model can generate relevant temporal and spatial event language from live video and chat history while also being relevant to chat history. |