Papers by Ziyue Zhang
Model Composition for Multimodal Large Language Models (2024.acl-long)
Copied to clipboard
Chi Chen, Yiyang Du, Zheng Fang, Ziyue Wang, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Maosong Sun, Yang Liu
| Challenge: | Existing methods for creating versatile MLLMs rely on joint training with paired instruction data, which is resource-intensive and challenging to extend to new modalities. |
| Approach: | They propose a new paradigm for multimodal large language models by reusing modality encoders and merging LLM parameters. |
| Outcome: | The proposed model retains the modal understanding capabilities of each original model. |
ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models (2025.acl-long)
Copied to clipboard
Ziyue Wang, Chi Chen, Fuwen Luo, Yurui Dong, Yuanchi Zhang, Yuzhuang Xu, Xiaolong Wang, Peng Li, Yang Liu
| Challenge: | Existing benchmarks for evaluating MLLMs have not addressed active perception . a novel benchmark is proposed to evaluate active perception in ML models . |
| Approach: | They propose a benchmark to evaluate active perception in Multimodal Large Language Models . they restrict the perceptual field of a model and require it to actively zoom or shift it . |
| Outcome: | The proposed benchmark focuses on a specialized form of Visual Question Answering (VQA) that eases and quantifies the evaluation yet challenging for existing MLLMs. |
Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024.acl-long)
Copied to clipboard
Rongjie Huang, Chunlei Zhang, Yongqi Wang, Dongchao Yang, Jinchuan Tian, Zhenhui Ye, Luping Liu, Zehan Wang, Ziyue Jiang, Xuankai Chang, Jiatong Shi, Chao Weng, Zhou Zhao, Dong Yu
| Challenge: | Large language models (LLMs) have been used for general-purpose interfaces across multiple tasks and languages. |
| Approach: | They propose to use large language models as a general-purpose interface across multiple tasks and languages. |
| Outcome: | The proposed model performs better on 200K hours of 6-language data for voice generation applications. |
Perspective Transition of Large Language Models for Solving Subjective Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized the field of natural language processing . performance of LLMs on subjective tasks is limited, authors say . |
| Approach: | They propose a method that allows LLMs to select between direct, role, and third-person perspectives for best way to solve corresponding subjective problem. |
| Outcome: | The proposed method outperforms widely used single fixed perspective based methods on 12 subjective tasks. |
Browse and Concentrate: Comprehending Multimodal Content via Prior-LLM Context Fusion (2024.acl-long)
Copied to clipboard
Ziyue Wang, Chi Chen, Yiqi Zhu, Fuwen Luo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Maosong Sun, Yang Liu
| Challenge: | Multimodal Large Language Models (MLLMs) lack understanding of multi-image and interleaved inputs due to the visual features encoded by frozen encoders before being fed into the LLM backbone. |
| Approach: | They propose a two phase paradigm to enable in-depth multimodal context fusion prior to feeding the features into LLMs. |
| Outcome: | The proposed paradigm boosts the performance on 7 multi-image scenarios, contributing to increments on average accuracy by 2.13% and 7.60% against strong MLLMs baselines with 3B and 11B LLMs, respectively. |
QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models (2025.emnlp-main)
Copied to clipboard
Jiajun Zhou, Yifan Yang, Kai Zhen, Ziyue Liu, Yequan Zhao, Ershad Banijamali, Athanasios Mouchtaris, Ngai Wong, Zheng Zhang
| Challenge: | Large Language Models (LLMs) are quantized to lower precision to reduce memory cost and latency in inference. |
| Approach: | They propose a quantized zeroth-order framework for fine-tuning Large Language Models (LLMs) using low-precision forward passes. |
| Outcome: | The proposed method achieves comparable results to first-order methods in FP8 and superior accuracy in INT8 and INT4 training. |
TCSinger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Control (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models fail to generate singing voices rich in stylistic nuances for unseen singers due to multifaceted nature of singing styles. |
| Approach: | They propose a zero-shot SVS model for style transfer across cross-lingual speech and singing styles and multi-level style control. |
| Outcome: | Experimental results show that TCSinger outperforms baseline models in synthesis quality, singer similarity, and style controllability. |
TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review (2025.emnlp-main)
Copied to clipboard
Yuan Chang, Ziyue Li, Hengyuan Zhang, Yuanbo Kong, Yanru Wu, Hayden Kwok-Hay So, Zhijiang Guo, Liya Zhu, Ngai Wong
| Challenge: | Large Language Models (LLMs) have shown significant potential in assisting peer review, but current methods struggle to generate thorough and insightful reviews while maintaining efficiency. |
| Approach: | They propose a framework that models paper review as a hierarchical and bidirectional question-answering process. |
| Outcome: | The proposed framework outperforms baselines on full review generation and actionable feedback comments generation tasks while reducing LLM token usage by up to 80% compared to computationally intensive approaches. |
Accelerating LLM Fine-Tuning via Embedding Knowledge Transfer (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on parameter-efficient fine-tuning (PEFT) have produced many state-of-the-art results by adapting LLMs to new tasks, but it requires substantial training data and time to enhance model performance. |
| Approach: | They propose a parameter-efficient fine-tuning framework which efficiently transfers knowledge from a small expert model to a target large model via embedding layers. |
| Outcome: | The proposed framework accelerates domain-specific fine-tuning, improves model performance and remains robust across diverse model families and PEFT methods. |
Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks focus on coarse-grained hallucination detection and fail to capture hallucinics . vision encoders exhibit unique hallucinian characteristics, but suboptimal of simple feature fusion. |
| Approach: | They propose a visual encoder that employs different training paradigms to instill inductive biases in visual encoded models. |
| Outcome: | The proposed system reduces hallucinations and improves model performance. |
Comprehensive Benchmarking of Long-Form Speech Generation in Diverse Scenarios (2026.findings-acl)
Copied to clipboard
Changhao Pan, Rui Yang, Han Wang, Zhuan Zhou, Xuming He, Wenxiang Guo, Ziyue Jiang, Ruiqi Li, Yu Zhang, Chenyuhao Wen, Ke Lei, Xiang Yin, Jingyu Lu, Zhiyuan Zhu, Zhou Zhao
| Challenge: | Existing evaluation benchmarks for long-form speech are limited to limited domains, creating a significant gap with the diverse downstream applications. |
| Approach: | They propose a benchmark that decomposes "long-form speech quality" into specific, disentangled dimensions. |
| Outcome: | The proposed benchmark decomposes “long-form speech quality” into specific, disentangled dimensions. |
TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Industry Systems (2024.emnlp-industry)
Copied to clipboard
Yilun Kong, Jingqing Ruan, YiHong Chen, Bin Zhang, Tianpeng Bao, Shi Shiwei, Du Qing, Xiaoru Hu, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, Xueqian Wang
| Challenge: | Large language models have demonstrated proficiency in addressing tasks that necessitate a combination of task planning and the usage of external tools. |
| Approach: | They propose a framework to enhance the task planning and tool usage abilities of LLMs in industrial systems. |
| Outcome: | The proposed framework enhances the task planning and tool usage abilities of LLM-based agents in industrial systems. |
MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria (2025.naacl-long)
Copied to clipboard
Wentao Ge, Shunian Chen, Hardy Chen, Nuo Chen, Junying Chen, Zhihong Chen, Wenya Xie, Shuo Yan, ChenghaoZhu ChenghaoZhu, Ziyue Lin, Dingjie Song, Xidong Wang, Anningzhe Gao, Zhang Zhiyi, Jianquan Li, Xiang Wan, Benyou Wang
| Challenge: | Existing evaluation methodologies for multimodal large language models are limited in evaluating objective queries without considering real-world user experiences. |
| Approach: | They propose to evaluate multimodal large language models with per-sample criteria using potent MLLM as the judge. |
| Outcome: | The proposed evaluation paradigm shows that it can be used to evaluate multimodal large language models with per-sample criteria. |
MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding (2026.acl-long)
Copied to clipboard
Fuwen Luo, Shengfeng Lou, Chi Chen, Ziyue Wang, Chenliang Li, Weizhou Shen, Jiyue Guo, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Yang Liu
| Challenge: | Existing methods for MLLMs struggle with fine-grained temporal reasoning . despite advances in video understanding, current methods struggle with time-sensitive tasks . |
| Approach: | They propose a time-stamp-aware multi-segment grounding method that enhances temporal understanding by introducing timestamps. |
| Outcome: | The proposed method outperforms existing methods on time-sensitive tasks and generalizes well across diverse temporal understanding scenarios. |
Versatile Framework for Song Generation with Prompt-based Control (2025.findings-emnlp)
Copied to clipboard
Yu Zhang, Wenxiang Guo, Changhao Pan, Zhiyuan Zhu, Ruiqi Li, Jingyu Lu, Rongjie Huang, Ruiyuan Zhang, Zhiqing Hong, Ziyue Jiang, Zhou Zhao
| Challenge: | Existing methods for song generation fail to generate vocals with prompt-based control and proper alignment. |
| Approach: | VersBand is a multi-task song generation framework for synthesizing high-quality songs with prompt-based control. |
| Outcome: | Experimental results show that VersBand performs better than baseline models across multiple song generation tasks. |
BrainLoc: Brain Signal-Based Object Detection with Multi-modal Alignment (2025.findings-emnlp)
Copied to clipboard
Jiaqi Duan, Xiaoda Yang, Kaixuan Luan, Hongshun Qiu, Weicai Yan, Xueyi Zhang, Youliang Zhang, Zhaoyang Li, Donglin Huang, JunYu Lu, Ziyue Jiang, Xifeng Yang
| Challenge: | BrainLoc is a lightweight object detection model guided by fMRI signals. |
| Approach: | They propose a brain-based object detection model guided by fMRI signals . they employ a multi-modal alignment strategy that enhances fmr feature extraction . |
| Outcome: | The proposed model improves fMRI-based object detection accuracy and convenience. |
TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis (2025.findings-acl)
Copied to clipboard
Yu Zhang, Wenxiang Guo, Changhao Pan, Dongyu Yao, Zhiyuan Zhu, Ziyue Jiang, Yuhan Wang, Tao Jin, Zhou Zhao
| Challenge: | Existing zero-shot singing voice synthesis models depend on phoneme and note boundary annotations, limiting their robustness and producing poor transitions between phonemes and notes. |
| Approach: | They propose a multi-task multilingual zero-shot SVS model with style transfer and style control based on various prompts. |
| Outcome: | Experimental results show that TCSinger 2 outperforms baseline models in subjective and objective metrics across multiple related tasks. |
Evolving Sparsity: Leveraging Token Importance Dynamics for Efficient LLM Decoding with Sparse Attention (2026.acl-long)
Copied to clipboard
| Challenge: | Efficient long-context inference remains a major challenge for large language models (LLMs), as the cost of attention computation during auto-regressive decoding grows linearly with the context length. |
| Approach: | They propose to model token importance as a dynamic process that evolves over decoding steps and propagates through model layers. |
| Outcome: | The proposed method outperforms baseline sparse attention methods and achieves speedups of up to 5.36 for attention latency and 2.33 for end-to-end decoding. |
GuiLoMo: Allocating Experts and Ranks for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors (2025.findings-emnlp)
Copied to clipboard
Xinrong Chen, Hengyuan Zhang, Yingmin Qiu, Xiao Liang, Ziyue Li, Guanyu Wang, Weiping Li, Tong Mo, Hayden Kwok-Hay So, Ngai Wong
| Challenge: | Low-Rank Adaptation (LoRA) methods are efficient for a large language model with reduced computational costs. |
| Approach: | They propose a layer-wise expert numbers and ranks allocation strategy with GuidedSelection Vectors. |
| Outcome: | The proposed method achieves superior or comparable performance to all baselines on three backbone models. |
CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation (2025.emnlp-main)
Copied to clipboard
Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Mingsong Yan, Zi Yang, Paul D. Hovland, Bogdan Nicolae, Franck Cappello, Sui Tang, Zheng Zhang
| Challenge: | Large foundation models have become huge, but they consume computational resources in pretraining. |
| Approach: | They propose to replace full-size layers with compute-efficient auto-encoders that enforce low-rank activations throughout training. |
| Outcome: | The proposed method reduces the computing cost by 2pmbtimes and improves training throughput by 1.86pmtime. |
A Survey on Foundation Language Models for Single-cell Biology (2025.acl-long)
Copied to clipboard
| Challenge: | Existing single-cell foundation language models are based on pre-trained and large language models. |
| Approach: | They review the development of single-cell foundation language models . they discuss data tokenization strategies and pre-training paradigms . |
| Outcome: | The proposed models have shown remarkable performance in a variety of single-cell data analysis tasks. |