CORD: Bridging the Audio–Text Reasoning Gap via Weighted On-policy Cross-modal Distillation (2026.findings-acl)
Copied to clipboard
Hu Jing, Danxiang Zhu, Xianlong Luo, Dan Zhang, Shuwei He, Yishu Lei, Shikun Feng, Hai-Tao Zheng, Jingzhou HE, Yu Sun, Hua Wu, Haifeng Wang
| Challenge: | Large Audio Language Models (LALMs) exhibit a degradation in knowledge and reasoning capabilities . empirical results show that CORD significantly bridges the audio–text performance gap . |
| Approach: | They propose a framework that performs online cross-modal self-distillation to bridge the acoustic-semantic gap between LALMs and text-based models. |
| Outcome: | The proposed framework bridges the acoustic-semantic gap between LALMs and text-based models . it employs on-policy reverse KL divergence with importance-aware weighting . |
Similar Papers
Closing the Modality Reasoning Gap for Speech Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Speech Large Language Models have a modality reasoning gap that is not addressed by prior work. |
| Approach: | They propose a reinforcement-learning framework that aligns text-conditioned and speech-conditioned trajectories through an asymmetric reward design. |
| Outcome: | Experiments on MMSU and OBQA show that the proposed framework narrows the modality reasoning gap and achieves state-of-the-art performance among 7B-scale Speech LLMs. |
Extending Audio Context for Long-Form Understanding in Large Audio-Language Models (2026.eacl-long)
Copied to clipboard
Yuatyong Chaichana, Pittawat Taveekitworachai, Warit Sirichotedumrong, Potsawee Manakul, Kunat Pipatanakul
| Challenge: | Prior work has introduced context-extension methods (e.g. YaRN) on unimodal LLMs, yet their application to LALMs remains unexplored. |
| Approach: | They propose a training-free, modality-decoupled extension method that modifies only audio token positions, leaving text positions intact to preserve the base LLM’s text capabilities. |
| Outcome: | The proposed method outperforms the original models across wide range of settings and provides significant performance improvement on long audio of unseen lengths. |
When Audio and Text Disagree: Revealing Text Bias in Large Audio-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Audio-Language Models (LALMs) are augmented with the ability to perceive audio, but their reliability when faced with conflicting inputs remains largely unexplored. |
| Approach: | They examine how LALMs prioritize information when presented with inconsistent audio-text pairs. |
| Outcome: | The proposed models display a significant bias toward textual input when presented with inconsistent audio-text pairs. |
Distilling LLM Reasoning into Dense Encoders: Bridging the Accuracy-Efficiency Gap in Recommendation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing distillation approaches target Small Language Models (SLMs) or Conventional Recommendation Models, but face a critical trade-off between computational cost and semantic reasoning capacity. |
| Approach: | They propose a framework that establishes a text encoder as the optimal student architecture for scalable recommendation. |
| Outcome: | Experiments on four datasets show that the proposed framework outperforms state-of-the-art models and achieves significantly reduced latency. |
Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | LSLMs have impressive conversational generation abilities, but consistently fall short of traditional pipeline systems on semantic understanding benchmarks. |
| Approach: | They propose to analyze the performance gap between speech and text inputs through a systematic experiment . they find that representation similarity is strongly correlated with the modality gap . |
| Outcome: | The proposed models improve the accuracy of speech inputs and their semantic understanding benchmarks. |
Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a promising solution for large language models, but their deployment remains computationally expensive. |
| Approach: | They propose a framework which iteratively balances training data within a fixed computational budget and enables the transfer of knowledge from expensive teacher LLMs to smaller student models. |
| Outcome: | The proposed framework achieves state-of-the-art performance across diverse long-tailed datasets, enhancing both the efficiency and efficacy of the distilled models. |
Distilling Long-CoT Reasoning through Collaborative Step-wise Multi-Teacher Decoding (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing curation-based approaches to inference are inefficient and fail to adapt dynamically, leading to redundant sampling and missed opportunities for complementary reasoning. |
| Approach: | They propose a collaborative multi-teacher decoding framework that performs step-wise reasoning synthesis guided by predictive perplexity–based scoring and beam search. |
| Outcome: | The proposed framework generates higher-quality reasoning data and achieves student-level results, demonstrating that fine-grained collaboration yields structured, efficient, and robust reasoning distillation. |
DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | Autoregressive (AR) large audio language models are expensive in data and computation . prior work shows diffusion-based LALMs can improve audio understanding under matched settings . |
| Approach: | They propose a diffusion-based LALM that upgrades the speech encoder and employs dual semantic and acoustic adapters. |
| Outcome: | a new model improves over existing autoregressive large language models and is competitive to strong AR models . the proposed model can make use of limited training data and improve inference efficiency . a recent study shows that diffusion-based models can improve audio understanding . |
MCC-KD: Multi-CoT Consistent Knowledge Distillation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable abilities in complex reasoning through chain of thought (CoT) prompting. |
| Approach: | They propose to generate multiple rationales for each question and enforce consistency among their predictions by minimizing the bidirectional KL-divergence between the answer distributions. |
| Outcome: | The proposed model achieves superior performance on in-distribution and commonsense reasoning benchmarks. |
Teaching Small Language Models Reasoning through Counterfactual Distillation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance in a wide range of downstream tasks. |
| Approach: | They propose a counterfactual distillation framework that leverages LLMs to generate high-quality counterfacts and utilizes multi-view CoT to enhance the diversity of reasoning samples. |
| Outcome: | The proposed framework enhances reasoning capabilities of large language models and is more robust to OOD data. |