Learning to See through Sound: From VggCaps to Multi2Cap for Richer Automated Audio Captioning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing AAC datasets suffer from short and simplistic captions, limiting expressiveness and semantic depth. |
| Approach: | They propose a multi-modal dataset that pairs audio with corresponding video and leverages large language models to generate rich, descriptive captions. |
| Outcome: | The proposed framework outperforms existing benchmarks in caption length, lexical diversity, and human-rated quality. |
Similar Papers
AudioCaps: Generating Captions for Audios in The Wild (N19-1)
Copied to clipboard
| Challenge: | a dataset of 46K audio clips with human-written text pairs is used to generate captions for audio . the task of translating a multimedia input source into natural language has been extensively studied over the past few years . |
| Approach: | They propose a top-down multi-scale encoder and aligned semantic attention for audio captioning. |
| Outcome: | The proposed captions are faithful to audio inputs and better than existing models. |
Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for audio captioning lack fine-grained detail and contextual accuracy due to limited unimodal or superficial information. |
| Approach: | They propose a two-stage automated pipeline that uses pretrained models to extract contextual cues from video . a large language model synthesizes these inputs to generate detailed and context-aware captions . |
| Outcome: | The proposed method is scalable and generates detailed and context-aware captions on large-scale audio datasets. |
X-ACE: Explainable and Multi-factor Audio Captioning Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics for automated audio captioning only provide an overall score . current evaluation checklists are inadequate to characterize the nuanced differences . |
| Approach: | They propose an explainable and multi-factor audio captioning evaluation paradigm . they define sound event, source, attribute and relation as four factors tailored for the audio description . |
| Outcome: | The proposed evaluation paradigm improves the quality of audio captions . it can detect mismatches and align with human perception, the authors show . |
Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologies (2025.findings-naacl)
Copied to clipboard
| Challenge: | Audio descriptions (ADs) are acoustic commentaries designed to assist blind and visually impaired individuals in accessing digital media content. |
| Approach: | They examine how state-of-the-art NLP and CV technologies can be applied to generate ADs . they identify essential research directions for the future . |
| Outcome: | The proposed technologies can be applied to generate audio descriptions (ADs) the process is time-consuming and costly, and requires significant human effort . the authors identify key research directions for the future . |
Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning general-purpose audio representations are limited in scope and coverage of audio attributes. |
| Approach: | They propose to use a 10.7M caption dataset to compare ALP with captioning . they find that contrastive learning yields competitive, transferable representations . |
| Outcome: | The proposed model yields competitive, transferable representations, while captioning exhibits better scalability. |
Enhancing Large Vision-Language Models with Ultra-Detailed Image Caption Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing pipelines for generating high-quality, ultra-detailed image captions are limited by the scarcity of image caption data. |
| Approach: | They propose a pipeline for generating high-quality, ultra-detailed image captions that integrates both pre-processing and post-processor stages. |
| Outcome: | The proposed pipeline improves LVLMs' perception and cognitive abilities across multiple vision-language benchmarks. |
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models for synthesising fluent, natural-sounding spoken audio captions do not require natural language text as an intermediate representation or source of supervision. |
| Approach: | They propose a model for directly synthesizing fluent, natural-sounding spoken audio captions for images that does not require natural language text as an intermediate representation or source of supervision. |
| Outcome: | The proposed model captures diverse visual semantics of images and can replace text with a set of discrete, sub-word speech units. |
ALCAP: Alignment-Augmented Music Captioner (2023.emnlp-main)
Copied to clipboard
| Challenge: | Traditional approaches to music captioning ignore the intricate interplay between the two . however, a comprehensive understanding of music necessitates the integration of both these elements. |
| Approach: | They propose a method to learn multimodal alignment between audio and lyrics through contrastive learning. |
| Outcome: | The proposed method achieves new state-of-the-art on two music captioning datasets. |
Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for audio-visual speech recognition use extra data to increase performance . a recent study shows that the use of unimodal self-supervised learning improves performance on multimodal tasks. |
| Approach: | They propose to use unimodal self-supervised learning to train AVSR models on unlabelled unilateral data. |
| Outcome: | The proposed model improves on lip reading sentences 2 by 30% even without an external language model. |
ICU: Conquering Language Barriers in Vision-and-Language Modeling by Dividing the Tasks into Image Captioning and Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that use multilingual captions for images have limited results due to the scarcity of training data. |
| Approach: | They propose a multilingual vision-and-language model that divides a V&L task into two stages . they propose IC, which takes the caption as the alt text and performs cross-lingual language understanding . |
| Outcome: | The proposed model can achieve state-of-the-art results for five languages and comparable results for the rest. |