Papers with CLAP
Do Audio-Language Models Understand Linguistic Variations? (2025.naacl-short)
Copied to clipboard
Ramaneswaran Selvakumar, Sonal Kumar, Hemant Kumar Giri, Nishit Anand, Ashish Seth, Sreyan Ghosh, Dinesh Manocha
| Challenge: | Existing open-vocabulary audio language models struggle to generalize to linguistic variations in textual queries. |
| Approach: | They propose a novel technique to learn audio-language representations agnostic to linguistic variations by reformulating contrastive loss used in CLAP architectures. |
| Outcome: | The proposed approach improves the performance of the open-vocabulary audio language models by 0.8%-13% across benchmarks and enhances robustness to linguistic variation. |
Guided by the Plan: Enhancing Faithful Autoregressive Text-to-Audio Generation with Guided Decoding (2026.eacl-long)
Copied to clipboard
| Challenge: | Autoregressive (AR) models excel at generating temporally coherent audio by producing tokens sequentially, yet they often falter in faithfully following complex textual prompts. |
| Approach: | They propose a lightweight auxiliary model trained with a GAE-inspired objective to predict final instruction-following quality from partial generations. |
| Outcome: | The proposed model achieves 10 points improvement in CLAP score over baseline AR models while maintaining computational parity with best-of-N decoding. |
On the Language Encoder of Contrastive Cross-modal Models (2024.findings-acl)
Copied to clipboard
Mengjie Zhao, Junya Ono, Zhi Zhong, Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Wei-Hsiang Liao, Takashi Shibuya, Hiromi Wakaki, Yuki Mitsufuji
| Challenge: | Pretrained audio-language models such as AudioCLIP and AudioCLAP have shown promising results on vision-language (VL) tasks. |
| Approach: | They extensively evaluate how unsupervised and supervised sentence embedding training affect language encoder quality and cross-modal task performance. |
| Outcome: | The proposed model improves on visual-language (VL) and audio-language tasks when the amount of training data is large. |
FineLAP: Taming Heterogeneous Supervision for Fine-grained Language-Audio Pretraining (2026.acl-long)
Copied to clipboard
| Challenge: | Existing audio-language models excel at clip-level understanding but struggle with frame-level tasks. |
| Approach: | They propose a novel training paradigm that advances both clip- and frame-level alignment in CLAP with heterogeneous data. |
| Outcome: | The proposed training paradigm improves both clip- and frame-level alignment in CLAP with heterogeneous data. |
Efficient AMR Parsing with CLAP: Compact Linearization with an Adaptable Parser (2024.lrec-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) parsers face efficiency challenges because of their large model size and computational time, which limit their accessibility within the research community. |
| Approach: | They propose a novel linearization system that simplifies encoding and reduces the number of tokens by between 40% and 50%. |
| Outcome: | The proposed system reduces the number of tokens by 40% and 50% while maintaining high performance while reducing training and inference times. |
PAT: Parameter-Free Audio-Text Aligner to Boost Zero-Shot Audio Classification (2025.naacl-long)
Copied to clipboard
| Challenge: | Audio-Language Models (ALMs) have demonstrated remarkable performance in zero-shot audio classification. |
| Approach: | They propose a training-free method that enhances audio and language representations using mutual feedback. |
| Outcome: | The proposed method outperforms vanilla zero-shot evaluation with significant margins of 0.42%-27.0%. |
Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval (2026.acl-long)
Copied to clipboard
| Challenge: | Experiments with AudioCaps, Clotho, and MECAT show that OEA achieves comparable text-to-text retrieval performance to state-of-the-art M2D-CLAP. |
| Approach: | They propose a retrieval-oriented encoder leveraging multimodal LLMs with native audio understanding that allows users to express their queries in five different ways. |
| Outcome: | Experiments on AudioCaps, Clotho, and MECAT show that OEA achieves comparable text-to-audio retrieval performance to state-of-the-art M2D-CLAP while demonstrating clear advantages in two critical areas. |
iKnow-audio: Integrating Knowledge Graphs with Audio-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Contrastive language-audio pretraining models learn by aligning audio and text in a shared embedding space. |
| Approach: | They propose a framework that integrates knowledge graphs with audio-language models to provide robust semantic grounding. |
| Outcome: | iKnow-audio improves disambiguation of acoustically similar sounds and reduces prompt engineering. |
FIGMA: Towards FIne-Grained Music retrievAl (2026.acl-long)
Copied to clipboard
| Challenge: | Existing music retrieval models fail to retrieve fine-grained musical attributes when using coarse semantic queries. |
| Approach: | They propose a multi-view contrastive architecture that captures high-level semantic context and fine-grained musical attributes within a unified representation space. |
| Outcome: | The proposed method outperforms existing CLAP-based music retrieval models on multiple benchmarks. |