Orchestrating Audio: Multi-Agent Framework for Long-Video Audio Synthesis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for video-to-audio dubbing for long-form content are fragmented and lack dedicated datasets. |
| Approach: | They propose a multi-agent framework that offers a coordinated, multi-component approach to long-video audio generation. |
| Outcome: | The proposed method outperforms state-of-the-art V2A models in audio quality. |
Similar Papers
MAViS: A Multi-Agent Framework for Long-Sequence Video Storytelling (2026.eacl-long)
Copied to clipboard
| Challenge: | Experimental results show that MAViS achieves state-of-the-art performance in assistive capability, visual quality, and video expressiveness. |
| Approach: | They propose an end-to-end multi-agent collaborative framework for long-sequence video storytelling that orchestrates specialized agents across multiple stages. |
| Outcome: | The proposed framework achieves state-of-the-art performance in assistive capability, visual quality, and video expressiveness. |
LongVideoAgent: Multi-Agent Reasoning with Long Videos (2026.acl-long)
Copied to clipboard
| Challenge: | a key emerging challenge is robust long video understanding, authors say . current methods compress content into lossy summaries or rely on limited toolsets . |
| Approach: | They propose a multi-agent framework where a master LLM coordinates a grounding agent and a vision agent to extract targeted textual observations. |
| Outcome: | The proposed model outperforms strong non-agent baselines on episode-level datasets . the proposed model significantly outperformed existing models on other datasets. |
Comprehensive Benchmarking of Long-Form Speech Generation in Diverse Scenarios (2026.findings-acl)
Copied to clipboard
Changhao Pan, Rui Yang, Han Wang, Zhuan Zhou, Xuming He, Wenxiang Guo, Ziyue Jiang, Ruiqi Li, Yu Zhang, Chenyuhao Wen, Ke Lei, Xiang Yin, Jingyu Lu, Zhiyuan Zhu, Zhou Zhao
| Challenge: | Existing evaluation benchmarks for long-form speech are limited to limited domains, creating a significant gap with the diverse downstream applications. |
| Approach: | They propose a benchmark that decomposes "long-form speech quality" into specific, disentangled dimensions. |
| Outcome: | The proposed benchmark decomposes “long-form speech quality” into specific, disentangled dimensions. |
PodAgent: A Comprehensive Framework for Podcast Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing automatic audio generation methods struggle to generate podcast-like audio programs effectively. |
| Approach: | They propose a framework for creating podcast-like audio programs that generates informative topic-discussion content by designing a multi-agent collaboration system, builds a voice pool and uses LLM-enhanced speech synthesis to generate expressive conversational speech. |
| Outcome: | The proposed framework surpasses direct GPT-4 generation in topic-discussion dialogue content, and produces more expressive conversational speech. |
UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions (2026.acl-long)
Copied to clipboard
Chunyu Qiang, Xiaopeng Wang, Kang Yin, Yuzhe Liang, Yuxin Guo, Teng Ma, Ziyu Zhang, Tianrui Wang, Cheng Gong, Yushen Chen, Ruibo Fu, Longbiao Wang, Jianwu Dang
| Challenge: | Generative audio modeling has been fragmented into specialized tasks such as text-to-speech (TTS), text- to-music (TTM), and text-ta (TTA) specialized models require reference audio for timbre cloning and strict phoneme alignment, whereas TTA models generate unstructured textures from open-ended captions. |
| Approach: | They propose a unified flow-matching framework capable of synthesizing speech, music, sound effects . they propose 'token injection mechanism' that projects unstructured environmental sounds into structured temporal latent space . |
| Outcome: | The proposed framework achieves state-of-the-art performance in instruction-based TTS and TTM while maintaining competitive fidelity in TTA. |
End-to-End Multilingual Automatic Dubbing via Duration-based Translation with Large Language Models (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Automatic dubbing (AD) aims to replace the original speech with translated speech that maintains precise temporal alignment (isochrony). |
| Approach: | They propose an end-to-end automatic dubbing framework that leverages large language models to integrate translation and timing control seamlessly. |
| Outcome: | The proposed framework achieves up to 24% relative gains on English, Spanish, and Korean language pairs while maintaining competitive translation quality measured by COMET scores. |
PresentAgent: Multimodal Agent for Presentation Video Generation (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Existing methods for generating static slides or text summaries are limited to producing narrated presentations. |
| Approach: | They propose a multimodal agent that transforms long-form documents into narrated presentations. |
| Outcome: | The present agent produces fully synchronized visual and spoken content that closely mimics human-style presentations. |
NexusSum: Hierarchical LLM Agents for Long-Form Narrative Summarization (2025.acl-long)
Copied to clipboard
| Challenge: | Summarizing long-form narratives requires capturing intricate plotlines, character interactions, and thematic coherence over tens of thousands of tokens. |
| Approach: | They propose a multi-agent LLM framework for narrative summarization that processes long-form text through a structured pipeline without fine-tuning. |
| Outcome: | The proposed framework achieves 30.0% improvement in BERTScore across books, movies, and TV scripts. |
AI4Reading: Chinese Audiobook Interpretation System Based on Multi-Agent Collaboration (2025.acl-demo)
Copied to clipboard
| Challenge: | Interpretative audiobooks are becoming more popular, but their manual creation process remains time-consuming and resource-intensive. |
| Approach: | They propose a multi-agent collaboration system that leverages large language models and speech synthesis technology to generate podcast-like audiobook interpretations. |
| Outcome: | The proposed system is open source and open to the public. |
Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluations of audio large language models focus on single audio inputs, but real-world applications often require processing multiple audio streams simultaneously. |
| Approach: | They propose a multi-audio evaluation benchmark that combines 20 audio inputs from 11 audio tasks to capture audio context. |
| Outcome: | The proposed model outperforms baseline models and achieves high data efficiency without human annotations. |