Papers by Ziyang Yu
VGA: Vision GUI Assistant - Minimizing Hallucinations through Image-Centric Fine-Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing Large Vision-Language Models (VLMs) often overly rely on internal text-based knowledge while neglecting visual inputs. |
| Approach: | They propose a model that balances attention image and text to enhance interpretation and reduce hallucinations by using a visual input. |
| Outcome: | The proposed model improves interpretation and reduces hallucinations by balancing attention image and text to enhance interpretation and reduction of hallucinosity. |
Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | End-to-end speech-to speech (S2S) dialogue systems face key challenges in incorporating external knowledge into their models. |
| Approach: | They propose a framework that directly retrieves relevant textual knowledge from speech queries. |
| Outcome: | The proposed framework improves the performance of end-to-end speech-tospeech dialogue systems while achieving higher retrieval efficiency. |
F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2025.acl-long)
Copied to clipboard
| Challenge: | Recent research in Text-to-Speech (TTS) has experienced great advancement . current models can synthesize speech for any given text and mimic the speaker of audio prompt. |
| Approach: | They propose a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer (DiT) without complex designs such as duration model, text encoder, and phoneme alignment, the text input is simply padded with filler tokens to the same length as input speech, and then denoising is performed for speech generation. |
| Outcome: | The proposed system achieves an inference RTF of 0.15, which is greatly improved compared to state-of-the-art diffusion-based models. |
Beyond Noise: Characterizing Creative Potential in Unverifiable LLM Hallucinations (2026.acl-long)
Copied to clipboard
Yu Yan, Chunhong Zhang, Haiyu Zhao, Ziyang Zeng, Zihao Liu, Yongkang Wu, Jianzhou Diao, YiJie Chen, Shujie Wang, Zheng Hu
| Challenge: | Large Language Models generate outputs that extend beyond established knowledge . prior work does not characterize the unverifiable space as a whole . |
| Approach: | They propose a novelty-verifiability characterization that distinguishes Creative Synthesis from Groundless Fabrication by a conceptual creation task. |
| Outcome: | The proposed model distinguishes Creative Synthesis (Region A) from Groundless Fabrication (Regium B) it shows that Region A is non-negligible and robust, persisting across generation strategies, models, domains, and embedding choices. |
Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite advances in reinforcement learning, data collection and fine-tuning remain costly and hard to scale. |
| Approach: | They propose a video-adaptive test-time scaling strategy that combines RL with a supervised fine-tuning strategy to improve video reasoning capability. |
| Outcome: | The proposed method surpasses existing models by 2.4% in accuracy using only 3.6% training samples. |
SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training (2025.findings-acl)
Copied to clipboard
Wenxi Chen, Ziyang Ma, Ruiqi Yan, Yuzhe Liang, Xiquan Li, Ruiyang Xu, Zhikang Niu, Yanqiao Zhu, Yifan Yang, Zhanxun Liu, Kai Yu, Yuxuan Hu, Jinyu Li, Yan Lu, Shujie Liu, Xie Chen
| Challenge: | a new spoken dialogue system with single-stage training is demonstrating its low latency and high quality . SLAM-Omni achieves zero-shot timbre control by modeling spoken language with semantic tokens . |
| Approach: | They propose a timbre-controllable, end-to-end voice interaction system with single-stage training. |
| Outcome: | The proposed system outperforms previous models on 4 GPUs with limited data. |
A Simple LLM Framework for Long-Range Video Question-Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | a recent study has shown that short video understanding is not trivial due to the need for long-range temporal reasoning capabilities. |
| Approach: | They propose a language-based short- and long-range question-answering framework LLoVi . they propose 'multi-round summarization prompt' that asks the LLM to summarize the captions . |
| Outcome: | The proposed framework outperforms the state-of-the-art on the EgoSchema dataset and to grounded VideoQA. |
Experience Retrieval-Augmentation with Electronic Health Records Enables Accurate Discharge QA (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve the reliability of Large Language Models (LLMs) in clinical applications require factual knowledge from open-ended datasets and clinical case-based knowledge to provide context grounded in real-world patient experiences. |
| Approach: | They propose a retrieval-augmented generation framework based on the electronic health record to offer contextual information from other patients’ discharge reports. |
| Outcome: | The proposed framework outperforms a text-based ranker in a clinical QA dataset with 1,280 discharge-related questions . |
GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement (2025.acl-long)
Copied to clipboard
Yifan Yang, Zheshu Song, Jianheng Zhuo, Mingyu Cui, Jinpeng Li, Bo Yang, Yexing Du, Ziyang Ma, Xunying Liu, Ziyuan Wang, Ke Li, Shuai Fan, Kai Yu, Wei-Qiang Zhang, Guoguo Chen, Xie Chen
| Challenge: | GigaSpeech 2 is a large-scale, multi-domain, multilingual speech recognition corpus for low-resource languages. |
| Approach: | They propose a large-scale, multi-domain, multilingual speech recognition corpus for low-resource languages and an automated pipeline for data crawling, transcription, and label refinement. |
| Outcome: | The proposed corpus reduces the word error rate for Thai, Indonesian, and Vietnamese on a realistic YouTube test set by 25% to 40% compared to Whisper large-v3. |
MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | MEXA is a training-free framework that performs modality- and task-aware aggregation of multiple expert models to enable effective multimodal reasoning across diverse domains. |
| Approach: | MEXA is a training-free framework that performs modality- and task-aware aggregation of multiple expert models. |
| Outcome: | MEXA performs modality- and task-aware aggregation of multiple expert models . it generates interpretable textual reasoning outputs and reasons over them using a Large Reasoning Model (LRM) MEX A consistently delivers performance improvements over strong multimodal benchmarks . |
VocalNet: Speech LLMs with Multi-Token Prediction for Faster and High-Quality Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show VocalNet outperforms existing open-source speech LLMs despite limited training data. |
| Approach: | They propose a scalable and model-agnostic training framework and a novel multi-token prediction paradigm for speech generation. |
| Outcome: | The proposed model outperforms open-source speech LLMs while outperforming existing open-sourced models. |
Evaluating the Expressive Appropriateness of Speech in Rich Contexts (2026.acl-long)
Copied to clipboard
Tianrui Wang, Ziyang Ma, Yizhou Peng, Haoyu Wang, Zhikang Niu, Zikang Huang, Yihao Wu, Yi-Wen Chao, Yu Jiang, Yuheng Lu, Guanrou Yang, Xuanchen Li, Hexin Liu, Chunyu Qiang, Cheng Gong, Yifan Yang, Tianchi Liu, Junyu Wang, Nana Hou, Meng Ge, Fuming You, Yang Wei, Zhongqian Sun, Hu Haifeng, Xiaobao Wang, Eng Siong Chng, Xie Chen, Longbiao Wang, Jianwu Dang
| Challenge: | Existing methods for evaluating expressive speech focus on word accuracy, naturalness, signal quality, or emotional intensity at the utterance level. |
| Approach: | They propose a framework for Evaluating Expressive Appropriateness in speech that assesses whether a speech sample aligns with the underlying communicative intent implied by its discourse-level narrative context. |
| Outcome: | The proposed framework outperforms existing speech evaluation and analysis systems on a human-annotated test set. |
UNO Arena for Evaluating Sequential Decision-Making Capability of Large Language Models (2024.emnlp-main)
Copied to clipboard
Zhanyue Qin, Haochuan Wang, Deyuan Liu, Ziyang Song, Cunhang Fan, Zhao Lv, Jinlin Wu, Zhen Lei, Zhiying Tu, Dianhui Chu, Xiaoyan Yu, Dianbo Sui
| Challenge: | Existing LLMs demonstrate powerful capabilities between tasks, but can they make sequential decisions? |
| Approach: | They propose to evaluate sequential decision-making capability of large language models (LLMs) using novel metrics based Monte Carlo methods. |
| Outcome: | The proposed benchmark improves sequential decision-making performance compared to the vanilla LLM player. |
URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lack of comprehensive evaluations for SDMs in speech-to-speech (S2S) scenarios is a major challenge for end-to end spoken dialogue models. |
| Approach: | They propose to provide an extensive evaluation framework for end-to-end spoken dialogue models (SDMs) that includes both cognitive dimensions and paralinguistic cues . |
| Outcome: | The proposed benchmark is divided into two difficulty levels: basic track and pro track, each comprising 20 test sets, evaluating the spoken dialogue model’s abilities in U**nderstanding, **R**easoning, and **O**ral conversation. |