Know your audience: specializing grounded language models with listener subtraction (2023.eacl-main)
Copied to clipboard
| Challenge: | Effective communication requires adapting to the idiosyncrasies of each communicative context. |
| Approach: | They propose a method for specializing grounded language models without supervision . they fine-tune an attention-based adapter between a CLIP vision encoder and a large language model . |
| Outcome: | The proposed method allows a speaker to adapt to the idiosyncracies of the listeners without supervision. |
Similar Papers
Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind (2023.findings-acl)
Copied to clipboard
| Challenge: | Adaptation is a process in human communication by which a speaker tunes its language to that of a listener to achieve communicative success. |
| Approach: | They propose a visual-based referential game between a knowledgeable speaker and a listener with limited visual and linguistic experience to model this adaptation mechanism. |
| Outcome: | The proposed model improves on plug-and-play approaches to controlled language generation without finetuning the speaker’s underlying language model. |
Countering Language Drift via Visual Grounding (D19-1)
Copied to clipboard
| Challenge: | Emergent multi-agent communication protocols are different from natural language . a long-standing goal of artificial intelligence research is to develop agents that can cooperate with other agents . |
| Approach: | They propose to use syntactic and semantic constraints to improve communication . they propose to combine these constraints with auxiliary training constraints to reduce language drift . |
| Outcome: | a new study shows that pre-trained agents retain English syntax while learning to convey intended meaning . the proposed training constraints can be used to mitigate language drift . |
Grounding Language in Multi-Perspective Referential Communication (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using a dataset of 2,970 human-written referring expressions, we find that the performance of automated models in both reference generation and comprehension lags behind that of pairs of human agents. |
| Approach: | They propose a task and dataset for referring expression generation and comprehension in multi-agent embodied environments where two agents must take into account one another's visual perspective to produce and understand references to objects in a scene. |
| Outcome: | The proposed model outperforms the strongest proprietary model and improves communicative success from 58.9 to 69.3% when trained with a listener. |
Calibrate your listeners! Robust communication-based training for pragmatic speakers (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work has investigated training NLP systems with communication-based objectives . prior work has focused on supervised learning, but is expensive to collect . |
| Approach: | They propose a method that uses a population of neural listeners to regularize speaker training. |
| Outcome: | The proposed method improves on ensemble- and dropout-based listening populations on reference games and generalizes to new games and listeners. |
DIXITWORLD: Evaluating Multimodal Abductive Reasoning in Vision-Language Models with Multi-Agent Dixit Gameplay (2026.acl-short)
Copied to clipboard
Yunxiang MO, Tianshi Zheng, Qing Zong, Jiayu Liu, Baixuan Xu, Yauwai Yim, Chunkit Chan, Jiaxin Bai, Yangqiu Song
| Challenge: | Existing evaluations of multimodal abductive reasoning are limited to static, single-agent tasks. |
| Approach: | They propose a multiagent evaluation suite that deconstructs the current evaluations of multimodal abductive reasoning in vision–language models. |
| Outcome: | The evaluation suite is based on two core components: DixitArena and DixitsBench. |
What Do Language Models Hear? Probing for Auditory Representations in Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | a linear probe is used to retrieve the correct text representation of an object given a snippet of audio related to that object. |
| Approach: | They develop a linear probe that retrieves the correct text representation of an object . they then test the probe's generalization to objects that were not seen during training . |
| Outcome: | The proposed model generalizes to objects that were not seen during training, the study finds . the model can learn representations of perceptual concepts that plausibly mirror the grounded representations . |
Context Shapes Emergent Communication about Concepts at Different Levels of Abstraction (2024.lrec-main)
Copied to clipboard
| Challenge: | Concept-level reference game allows speakers to communicate concepts at different levels of abstraction and in different contexts. |
| Approach: | They use a symbolic dataset that disentangles concept type and context to study the influence of these factors on the emerging language. |
| Outcome: | The proposed model disentangles concept type and context to study the communication of concepts at different levels of abstraction and in different contexts. |
Agents generalize to novel levels of abstraction by using adaptive linguistic strategies (2025.findings-acl)
Copied to clipboard
| Challenge: | Abstract: Abstracts are fundamental to building well-generalizing models. |
| Approach: | They propose to use a concept-level reference game to generalize concepts . they find that agents can learn robust concepts based on which they can generalize . |
| Outcome: | The proposed model can generalize from generic to very specific concepts, while reusing many messages from training. |
Emergent Communication Pretraining for Few-Shot Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | state-of-the-art models that rely on multilingual pretrained encoders achieve sample efficiency in downstream applications, but lack abundant amounts of unlabelled text. |
| Approach: | They propose a method to pretrain neural networks via emergent communication from referential games by grounding communication on images as a crude approximation of real-world environments. |
| Outcome: | The proposed method significantly improves machine translation in few-shot settings and provides an evaluation protocol to probe the properties of emergent languages ex vitro. |
SpidR-Adapt: A Universal Speech Representation Model for Few-Shot Adaptation (2026.acl-long)
Copied to clipboard
Mahi Luthra, Jiayi Shen, Maxime Poli, Angelo Ortiz Tandazo, Yosuke Higuchi, Youssef Benchekroun, Martin Gleize, Charles-Éric Saint-James, Dongyan Lin, Phillip Rust, Angel Villar-Corrales, null Surya, Vanessa Stark, Rashel Moritz, Juan Pino, Yann LeCun, Emmanuel Dupoux
| Challenge: | Empirically, SpidR-Adapt achieves rapid gains in phonemic discriminability and downstream spoken language modeling scores . current self-supervised learning models require thousands of hours of training data to learn meaningful linguistic representations. |
| Approach: | They propose a bi-level optimization framework for rapid adaptation of speech units to new languages using minimal unlabeled data. |
| Outcome: | The proposed model achieves rapid gains in phonemic discriminability and spoken language modeling scores . it surpasses in-domain toplines after training on less than 1h of target-language audio . |