ChatMusician: Understanding and Generating Music Intrinsically with LLM (2024.findings-acl)
Copied to clipboard
Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, Liumeng Xue, Ziyang Ma, Qin Liu, Tianyu Zheng, Yizhi Li, Yinghao Ma, Yiming Liang, Xiaowei Chi, Ruibo Liu, Zili Wang, Chenghua Lin, Qifeng Liu, Tao Jiang, Wenhao Huang, Wenhu Chen, Jie Fu, Emmanouil Benetos, Gus Xia, Roger Dannenberg, Wei Xue, Shiyin Kang, Yike Guo
| Challenge: | Despite LLMs' impressive capabilities in musical knowledge, music reasoning remains an unsolved task. |
| Approach: | They propose an open-source large language model (LLM) that integrates intrinsic musical abilities into LLaMA2 and GPT-3.5. |
| Outcome: | The proposed model can understand and generate music with a pure text tokenizer without external multi-modal neural structures or tokenizers. |
Similar Papers
MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Models (2023.emnlp-demo)
Copied to clipboard
| Challenge: | MusicAgent integrates numerous music-related tools and an autonomous workflow to address user requirements. |
| Approach: | a new system is built to integrate music-related tools and an autonomous workflow . the system is based on large language models (LLMs) that can be used to organize and decompose requests . |
| Outcome: | the proposed system integrates numerous music-related tools and an autonomous workflow to address user requirements. |
SongComposer: A Large Language Model for Lyric and Melody Generation in Song Composition (2025.acl-long)
Copied to clipboard
Shuangrui Ding, Zihan Liu, Xiaoyi Dong, Pan Zhang, Rui Qian, Junhao Huang, Conghui He, Dahua Lin, Jiaqi Wang
| Challenge: | Creating lyrics and melodies in symbolic format requires expert knowledge of melody and an advanced understanding of lyrics. |
| Approach: | They introduce SongComposer, a music-specialized large language model that can create symbolic lyrics and melodies following instructions. |
| Outcome: | The proposed model outperforms existing models in symbolic song composition tasks. |
Thesis Proposal: Multimodal Benchmark for Music Understanding in Large Language Models (2026.eacl-srw)
Copied to clipboard
| Challenge: | Existing music-focused benchmarks are fragmented, largely single-modality, Western-centric . existing methods for evaluating MLLMs are lacking reproducibility and reliability . |
| Approach: | They propose to develop a musically multimodal benchmark that will integrate music into the benchmark. |
| Outcome: | The proposed benchmark will integrate culturally diverse musical material beyond the dominant Western canon. |
The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks to evaluate LLMs' capabilities are inadequate for assessing their musical capabilities. |
| Approach: | They propose to use a large-scale music benchmark specifically designed to evaluate the music-related capabilities of large language models (LLMs). |
| Outcome: | The proposed framework evaluates 16 large language models in the domain of music. |
The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing instruction tuned large language models (LLMs) struggle to understand cross-lingual sociopragmatic meaning (SM) lack of comprehensive investigation into their ability to understand SM is partly due to SM not being adequately represented in any of the existing benchmarks. |
| Approach: | They evaluate the performance of instruction tuned large language models (LLMs) on a multilingual benchmark specifically designed for SM understanding. |
| Outcome: | The proposed benchmark outperforms instruction tuned large language models on a wide range of tasks but falls behind task-specific finetuned models. |
Toward Beginner-Friendly LLMs for Language Learning: Controlling Difficulty in Conversation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Practicing conversations with large language models is a promising alternative to traditional in-person language learning. |
| Approach: | They propose a new token-level evaluation metric, Token Miss Rate, that measures the proportion of incomprehensible tokens per utterance and correlates strongly with human judgments. |
| Outcome: | The proposed methods improve comprehensibility for beginner speakers from 39.4% to 83.3%, compared with prompting alone and a token-level evaluation metric, Token Miss Rate (TMR). |
LLM Agents Implement an NLG System from Scratch: Building Interpretable Rule-Based RDF-to-Text Generators (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing neural approaches to generate RDF-to-text are limited in their implementation. |
| Approach: | They propose a framework where the model is “trained” through collaborative interactions among multiple LLM agents rather than traditional backpropagation. |
| Outcome: | The proposed framework reduces hallucinations and fluency penalties on the WebNLG and OpenDialKG datasets. |
LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient (2026.acl-long)
Copied to clipboard
Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Using generic and efficient benchmark generators, human annotators are limited by inefficiency . current benchmark generator methods rely on seed signals, leading to long cycles and high costs . |
| Approach: | They propose a framework to evaluate LLMs as generic benchmark generators and integrate them as BenchMaker. |
| Outcome: | The proposed framework achieves comparable performance to human-annotated benchmarks on most metrics. |
DeepResonance: Enhancing Multimodal Music Understanding via Music-centric Multi-way Instruction Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in music large language models have significantly improved music understanding tasks, but the potential of incorporating additional modalities such as images, videos and textual music features remains unexplored. |
| Approach: | They propose a multimodal music understanding LLM fine-tuned via multi-way instruction tuning with multi-ways aligned music, text, image, and video data. |
| Outcome: | The proposed model achieves state-of-the-art performance across six music understanding tasks and zero-shot scenarios. |
Musical Score Understanding Benchmark: Evaluating Large Language Models’ Comprehension of Complete Musical Scores (2026.acl-long)
Copied to clipboard
Congren Dai, Yue Yang, Krinos Li, Huichi Zhou, Shijie Liang, Zhang Bo, Enyang Liu, Ge Jin, Hongran An, Haosen Zhang, Peiyuan Jing, KinHei Lee, Zhenxuan Zhang, Xiaobing Li, Maosong Sun
| Challenge: | Existing benchmarks for musical score understanding are narrow in scope, focusing on isolated fragments, short excerpts, or multiple-choice formulations, rather than supporting holistic reasoning over entire scores. |
| Approach: | They propose a benchmark for score-level musical understanding across textual and visual modalities. |
| Outcome: | The musical score understanding benchmark contains 1,800 question-answer pairs from works by Bach, Beethoven, Chopin, Debussy, and others. |