I see what you mean: Co-Speech Gestures for Reference Resolution in Multimodal Dialogue (2025.findings-acl)
Copied to clipboard
| Challenge: | Using representational co-speech gestures, face-to-face interaction participants resolve references to objects using speech and gestures. |
| Approach: | They propose a multimodal reference resolution task centred on representational gestures . they propose 'self-supervised' pre-training approach to gesture representation learning that grounds body movements in spoken language. |
| Outcome: | The proposed approach aligns with expert annotations and has significant predictive power. |
Similar Papers
Disambiguating Reference in Visually Grounded Dialogues through Joint Modeling of Textual and Multimodal Semantic Structures (2025.acl-long)
Copied to clipboard
| Challenge: | incorporating textual and multimodal reference resolution improves performance in visual-based reference resolution . Phrase grounding is a well-established task for understanding semantic relations between mentions and objects . ambiguities caused by pronouns and ellipses can arise in visually grounded dialogues . |
| Approach: | They propose a framework that unifies textual and multimodal reference resolution by mapping mention embeddings to object embeddements and selecting mentions or objects based on their similarity. |
| Outcome: | The proposed framework performs better in phrase grounding than other models for this task. |
A Formal Analysis of Multimodal Referring Strategies Under Common Ground (2020.lrec-1)
Copied to clipboard
| Challenge: | a recent study has focused on multimodality in the CL/NLP community, but it has not been widely studied. |
| Approach: | They propose to analyze mixed-modality definite referring expressions using gestures and linguistic descriptions. |
| Outcome: | The proposed models can predict viewer judgment of referring expressions and generate more natural and informative expressions. |
Towards Understanding the Relation between Gestures and Language (2022.coling-1)
Copied to clipboard
| Challenge: | a new study explores the relationship between gestures and language . we use contrastive learning to learn gesture embeddings . |
| Approach: | They adapt a semi-supervised multimodal model to learn gesture embeddings using Ted talks . they show gestures are predictive of the native language of the speaker . |
| Outcome: | The proposed model learns gesture embeddings from a multimodal dataset . it shows that gesture embeds are predictive of the native language of the speaker . |
Encoding Gesture in Multimodal Dialogue: Creating a Corpus of Multimodal AMR (2024.lrec-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) was designed to represent sentence meaning in English text, but recent research has explored its adaptation to broader domains, including documents, dialogues, spatial information, cross-lingual tasks, and gesture. |
| Approach: | They propose to annotate a multimodal (speech and gesture) AMR corpus in a task-based setting and capture coreference relationships across modalities. |
| Outcome: | The proposed corpus captures coreference relationships across modalities, enabling fine-grained analysis of how gesture and natural language interact. |
Modeling Turn-Taking with Semantically Informed Gestures (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing computational models of turn-taking relied on verbal cues and prosody. |
| Approach: | They propose a framework that integrates text, audio, and gestures to model multimodal turn-taking using semantic annotations. |
| Outcome: | The proposed framework shows that incorporating semantically guided gestures yields consistent performance gains over baselines. |
J-CRe3: A Japanese Conversation Dataset for Real-world Reference Resolution (2024.lrec-main)
Copied to clipboard
Nobuhiro Ueda, Hideko Habe, Akishige Yuguchi, Seiya Kawano, Yasutomo Kawanishi, Sadao Kurohashi, Koichiro Yoshino
| Challenge: | Existing studies have ground referential expressions in language to real-world objects for cooperative action generation. |
| Approach: | They propose a Japanese Conversation dataset for real-world reference resolution that ground referential expressions to visual information observed in egocentric views. |
| Outcome: | The proposed dataset contains egocentric video and dialogue audio of real-world conversations between two people acting as a master and assistant robot at home. |
How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent advances in foundation models have sparked growing interest in expanding their text processing capabilities to speech. |
| Approach: | They analyze the model activations from semantically equivalent sentences across languages in the text and speech modalities and examine how text and spoken are represented in recent multimodal foundation models. |
| Outcome: | The proposed models exhibit cross-lingual differences, but are not explicitly trained for modality-agnostic representations. |
A Corpus of Natural Multimodal Spatial Scene Descriptions (L18-1)
Copied to clipboard
| Challenge: | Existing work on multimodal spatial descriptions combines speech and hand gestures to form a corpus of multimodal descriptions. |
| Approach: | They present a corpus of multimodal spatial descriptions as commonly occurring in route giving tasks. |
| Outcome: | The proposed corpus of multimodal spatial descriptions is more amenable to computational analysis and useable for learning natural computer interfaces. |
Which One Are You Referring To? Multimodal Object Identification in Situated Dialogue (2023.eacl-srw)
Copied to clipboard
| Challenge: | a demand for multimodal dialogue systems is increasing for situated dialogues, where a dialogue agent shares a co-observed vision or physical space with the user. |
| Approach: | They propose three methods to solve multimodal object identification problem using situated dialogue dataset SIMMC 2.1. |
| Outcome: | The proposed method improves by 20% F1-score on the largest situated dialogue dataset, SIMMC 2.1. |
Grounding Language in Multi-Perspective Referential Communication (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using a dataset of 2,970 human-written referring expressions, we find that the performance of automated models in both reference generation and comprehension lags behind that of pairs of human agents. |
| Approach: | They propose a task and dataset for referring expression generation and comprehension in multi-agent embodied environments where two agents must take into account one another's visual perspective to produce and understand references to objects in a scene. |
| Outcome: | The proposed model outperforms the strongest proprietary model and improves communicative success from 58.9 to 69.3% when trained with a listener. |