Papers by Haizhou Li

Modeling Code-Switch Languages Using Bilingual Parallel Corpus (2020.acl-main)

Challenge:	Existing end-to-end speech large language models rely on large-scale annotated data for training, while data-efficient training has not been discussed in depth.
Approach:	They propose a training strategy and a novel architecture to address representation space gap and sequence length inconsistency in speech and text.
Outcome:	The proposed model outperforms other advanced speech LLMs in speech translation and AIR-Bench speech tasks with only a fraction of the training data.

Copied to clipboard

Grandee Lee, Haizhou Li

Multilinguality

Challenge:	Existing models for bilingual language modeling are limited due to lack of training data and syntactic structure.
Approach:	They propose a bilingual attention language model that performs language modeling objective with a quasi-translation objective to model the monolingual and cross-lingual sequential dependency.
Outcome:	The proposed model reduces the perplexity of 20.5% over the best-reported model.

From Word to World: Evaluate and Mitigate Culture Bias in LLMs via Word Association Test (2025.emnlp-main)

Copied to clipboard

Xunlian Dai, Li Zhou, Benyou Wang, Haizhou Li

Challenge:	Multilingual and cross-cultural WAT reveal how culture modulates perceptual and interactive patterns.
Approach:	They propose to embed cultural-specific semantic associations directly within large language models (LLMs) to address cultural preference.
Outcome:	The proposed model significantly improves cross-cultural alignment, capturing diverse semantic associations.

S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Models (2026.acl-long)

Copied to clipboard

Feng Jiang, Zhiyu Lin, Yiyang Liu, Liumeng Xue, Fan Bu, Yuhao Du, Xiangying Chen, Benyou Wang, Haizhou Li

Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs (2026.acl-long)

Challenge:	Existing benchmarks rely heavily on text-based evaluation and largely ignore paralinguistic cues such as prosody, emotion, and speaker traits.
Approach:	They propose a speech-native benchmark for evaluating instruction-following S2S models with explicit assessment of both semantic understanding and paralinguistic expression.
Outcome:	The proposed system enables more natural, robust, and human-aligned speech agents.

Copied to clipboard

Zhenyu Liu, Xuanyu Zhang, Yunxin li, Qixun Teng, Shenyuan Jiang, Haolan Chen, Mingjun Zhao, Fanbo Meng, Yu Xu, Yancheng He, Baotian Hu, Haizhou Li, Min Zhang

DynaEval: Unifying Turn and Dialogue Level Evaluation (2021.acl-long)

Challenge:	despite significant progress, full-duplex SLMs are constrained by severe modality interference, authors say . modality interferes with acoustic and semantic modeling, making them unintelligent and unnatural . authors propose a hierarchical parameter separation strategy that decouples conflicting modalities in deep layers .
Approach:	They propose a hierarchical parameter separation strategy that decouples conflicting modalities in deep layers while preserving cross-modality coherence via a dedicated semantic alignment channel.
Outcome:	The proposed method significantly advances the state of the art on full-duplex benchmarks . it decouples conflicting modalities in deep layers while preserving cross-modality coherence .

Copied to clipboard

Chen Zhang, Yiming Chen, Luis Fernando D’Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, Haizhou Li

Natural Language Interfaces

Challenge:	Existing evaluation metrics focus on the turn-level quality of a dialogue . a unified framework that holistically considers the quality of the entire dialogue is needed .
Approach:	They propose a unified automatic evaluation framework which holistically considers the quality of the entire dialogue.
Outcome:	The proposed framework outperforms the state-of-the-art dialogue coherence model and correlates strongly with human judgements across multiple evaluation aspects at both turn and dialogue level.

DynaThink: Fast or Slow? A Dynamic Decision-Making Framework for Large Language Models (2024.emnlp-main)

Copied to clipboard

Jiabao Pan, Yan Zhang, Chen Zhang, Zuozhu Liu, Hongwei Wang, Haizhou Li

Does Mapo Tofu Contain Coffee? Probing LLMs for Food-related Cultural Knowledge (2025.naacl-long)

Challenge:	Large language models (LLMs) have emerged as prominent foundation models for diverse applications due to their outstanding ability to understand and generate humanlike text.
Approach:	They propose a dynamic decision-making framework that categorizes tasks into two distinct pathways: 'Fast' and 'Slow' they propose 'self-consistency' strategy to replace the straight-forward decoding method used in COT prompting .
Outcome:	The proposed method achieves more than 3% increase in accuracy with lower cost on five popular reasoning benchmarks.

Copied to clipboard

Li Zhou, Taelin Karidi, Wanlong Liu, Nicolas Garneau, Yong Cao, Wenyu Chen, Haizhou Li, Daniel Hershcovich

Multilinguality

Challenge:	Recent studies have highlighted the presence of cultural biases in Large Language Models (LLMs), yet lack a robust methodology to dissect these phenomena comprehensively.
Approach:	They propose a multilingual dataset centered on food-related cultural facts and variations in food practices.
Outcome:	The proposed model incorporates cultural context significantly and improves its ability to access cultural knowledge.

Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profiles (2025.acl-long)

Copied to clipboard

Kuang Wang, Xianfei Li, Shenghao Yang, Li Zhou, Feng Jiang, Haizhou Li

Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework (2022.emnlp-main)

Challenge:	Existing user simulators lack authenticity and user-level diversity in interactions with large language models.
Approach:	They propose a user simulator with implicit user profiles that infers user profiles from human-machine interactions to simulate personalized and realistic dialogues.
Outcome:	The proposed framework outperforms baselines in authenticity and diversity while maintaining comparable consistency.

Copied to clipboard

Yiming Chen, Yan Zhang, Bin Wang, Zuozhu Liu, Haizhou Li

Responsible & Trustworthy NLP

Challenge:	Existing supervised sentence embedding techniques rely on expensive human-annotated sentence pairs as the supervised signals.
Approach:	They propose a semi-supervised sentence embedding framework that leverages large-scale unlabeled data.
Outcome:	The proposed framework surpasses state-of-the-art methods on four domain adaptation tasks.

TS-Align: A Teacher-Student Collaborative Framework for Scalable Iterative Finetuning of Large Language Models (2024.findings-emnlp)

Copied to clipboard

Chen Zhang, Chengguang Tang, Dading Chong, Ke Shi, Guohua Tang, Feng Jiang, Haizhou Li

xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark (2023.findings-emnlp)

Challenge:	Mainstream approaches to aligning large language models heavily rely on human preference data.
Approach:	They propose a framework that fine-tunes a policy model using pairwise feedback data automatically mined from its outputs.
Outcome:	The proposed framework outperforms the base model with an average win rate of 69.7% across seven conversational or instruction-following datasets.

Copied to clipboard

Chen Zhang, Luis D’Haro, Chengguang Tang, Ke Shi, Guohua Tang, Haizhou Li

Natural Language Interfaces

Challenge:	Currently, human evaluation is the most reliable way to holistically judge the quality of the dialogue.
Approach:	They propose to use English dialogue evaluation metrics to generalize them to other languages.
Outcome:	The proposed metrics outperform OpenAI’s ChatGPT in terms of average Pearson correlations over all datasets and languages.

Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models (2025.naacl-long)

Copied to clipboard

Ziche Liu, Rui Ke, Yajiao Liu, Feng Jiang, Haizhou Li

Unveiling the Achilles’ Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models (2024.findings-acl)

Challenge:	Existing studies focus on data selection but lack a clear, unified framework . variability in experimental settings complicates systematic comparisons .
Approach:	They propose a three-stage scheme to standardize data selection for fine-tuning large language models . they propose unified comparison approach that incorporates ratio-based efficiency and ranking-based feasibility metrics to address inconsistencies across experiments.
Outcome:	The proposed scheme outperforms existing methods in a dozen key studies and identifies key challenges.

Copied to clipboard

Yiming Chen, Chen Zhang, Danqing Luo, Luis Fernando D’Haro, Robby Tan, Haizhou Li

Robustness in NLP

Challenge:	Recent studies have highlighted various neural metrics that align well with human evaluations.
Approach:	They propose a black-box adversarial framework that generates strong disagreements between human and victim evaluators.
Outcome:	The proposed framework can significantly improve the performance of human and victim evaluators.

Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation (2025.emnlp-main)

Copied to clipboard

Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, Baishakhi Ray

HuatuoGPT, Towards Taming Language Model to Be a Doctor (2023.findings-emnlp)

Challenge:	In the era of evaluating large language models, data contamination is an increasingly prominent concern . static benchmarking has been used for evaluation, but there are limitations of dynamic benchmarks .
Approach:	They propose a series of optimal design principles for dynamic benchmarking and analyze the limitations of existing static benchmarks.
Outcome:	The proposed benchmarks highlight a critical gap in the evaluation of LLMs.

Copied to clipboard

Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhang Zhiyi, Qingying Xiao, Xiang Wan, Benyou Wang, Haizhou Li

CrossTune: Black-Box Few-Shot Classification with Label Enhancement (2024.lrec-main)

Challenge:	Experimental results show that the distilled language model outperforms its teacher model (ChatGPT) in most cases.
Approach:	They propose a Large Language Model (LLM) that leverages both distilled data from ChatGPT and real-world data fromdoctors in the supervised fine-tuning stage.
Outcome:	The proposed model outperforms the teacher model in most cases by using additional real-world data and RLMF to align the language model with the merits of both sources.

Copied to clipboard

Danqing Luo, Chen Zhang, Yan Zhang, Haizhou Li

Multimodal Fine-grained Context Interaction Graph Modeling for Conversational Speech Synthesis (2025.emnlp-main)

Challenge:	Training large-scale language models requires substantial computation resources . current research focuses on adapting black-box models to downstream tasks using prompt optimization .
Approach:	They propose a label-enhanced cross-attention network called CrossTune to improve the generalization of the model.
Outcome:	The proposed approach outperforms the state-of-the-art black-box tuning method by 5.7% on average.

Copied to clipboard

Zhenqi Jia, Rui Liu, Berrak Sisman, Haizhou Li

Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion (2025.acl-long)

Challenge:	Existing methods overlook the fine-grained semantic and prosodic interaction modeling at the word level.
Approach:	They propose a novel approach to generate conversational prosody by understanding multimodal dialogue history (MDH) using fine-grained semantic and prosodic interaction modeling, they construct specialized multimodal fine-grain dialogue interaction graphs that encode interaction between word-level semantics and prosody.
Outcome:	The proposed system outperforms baseline models in terms of prosodic expressiveness.

Copied to clipboard

Jianqing Zhu, Huang Huang, Zhihang Lin, Juhao Liang, Zhengyang Tang, Khalid Almubarak, Mosen Alharthi, Bang An, Juncai He, Xiangbo Wu, Fei Yu, Junying Chen, Ma Zhuoheng, Yuhao Du, He Zhang, Saied Alshahrani, Emad A. Alghamdi, Lian Zhang, Ruoyu Sun, Haizhou Li, Benyou Wang, Jinchao Xu

Bootstrapped Unsupervised Sentence Representation Learning (2021.acl-long)

Challenge:	In the evolving landscape of large language models, the predominant focus has been on English and Chinese.
Approach:	They propose to utilize Arabic-specific vocabulary in the tokenizer to accelerate decoding.
Outcome:	The proposed model achieves decent performance comparable to the best Arabic LLMs across various Arabic benchmarks.

Copied to clipboard

Yan Zhang, Ruidan He, Zuozhu Liu, Lidong Bing, Haizhou Li

Representation Learning

Challenge:	Existing approaches to learn sentence representations rely on quality labeled data.
Approach:	They propose a Siamese Network which maximizes similarity between two augmented views of each sentence.
Outcome:	The proposed method outperforms state-of-the-art methods on STS and classification tasks.

Dynamic Transformers Provide a False Sense of Efficiency (2023.acl-long)

Copied to clipboard

Yiming Chen, Simin Chen, Zexin Li, Wei Yang, Cong Liu, Robby Tan, Haizhou Li

Responsible & Trustworthy NLP

Challenge:	Pre-trained language models typically lead to high computational cost during inference.
Approach:	They propose a slowdown attack framework that can reduce inference efficiency by 80% by leveraging existing adversarial attacks targeting model accuracy.
Outcome:	The proposed framework can reduce the efficiency of multi-exit models by 80% on average, validating its effectiveness and generalization ability.

Just Rank: Rethinking Evaluation with Word and Sentence Similarities (2022.acl-long)

Copied to clipboard

Bin Wang, C.-C. Jay Kuo, Haizhou Li

UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity Mixture-of-Experts (2026.acl-long)

Challenge:	Word and sentence similarity tasks are the de facto evaluation method for embeddings.
Approach:	They propose a new intrinsic evaluation method called EvalRank which shows a much stronger correlation with downstream tasks.
Outcome:	The proposed method shows a much stronger correlation with downstream tasks and is released for future benchmarking purposes.

Copied to clipboard

Zhenyu Liu, Yunxin li, Xuanyu Zhang, Qixun Teng, Shenyuan Jiang, Xinyu Chen, Haoyuan Shi, Haolan Chen, Fanbo Meng, Mingjun Zhao, Yu Xu, Yancheng He, Baotian Hu, Haizhou Li, Min Zhang

Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Models (2024.findings-emnlp)

Challenge:	Recent advances in unified multimodal models indicate a clear trend towards comprehensive content generation.
Approach:	They propose a unified speech and music generation model built upon a novel framework . they propose specialized MoE architectures and curated training strategies to tackle data imbalances .
Outcome:	The proposed model achieves state-of-the-art performance on major speech and music generation benchmarks.

Copied to clipboard

Yiming Chen, Xianghu Yue, Xiaoxue Gao, Chen Zhang, Luis Fernando D’Haro, Robby Tan, Haizhou Li

FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022.emnlp-main)

Challenge:	Existing evaluations of audio large language models focus on single audio inputs, but real-world applications often require processing multiple audio streams simultaneously.
Approach:	They propose a multi-audio evaluation benchmark that combines 20 audio inputs from 11 audio tasks to capture audio context.
Outcome:	The proposed model outperforms baseline models and achieves high data efficiency without human annotations.

Copied to clipboard

Chen Zhang, Luis Fernando D’Haro, Qiquan Zhang, Thomas Friedrichs, Haizhou Li

Chain-Talker: Chain Understanding and Rendering for Empathetic Conversational Speech Synthesis (2025.findings-acl)

Challenge:	Recent model-based reference-free metrics for open-domain dialogue evaluation lack correlations with human judgment and poor interpretability.
Approach:	They propose a multi-dimensional dialogue-level metric with three sub-metrics targeting a specific dimension.
Outcome:	The proposed metric outperforms existing models and sub-metrics in three high-quality dialogue evaluation benchmarks.

Copied to clipboard

Yifan Hu, Rui Liu, Yi Ren, Xiang Yin, Haizhou Li

Advancing Topic Segmentation and Outline Generation in Chinese Texts: The Paragraph-level Topic Representation, Corpus, and Benchmark (2024.lrec-main)

Challenge:	Current generative CSS models face interpretability limitations due to insufficient emotional perception and redundant discrete speech coding.
Approach:	They propose a framework that aligns synthesized speech with the emotional context of user-agent interactions to achieve empathy.
Outcome:	The proposed framework produces more expressive speech than existing methods on three datasets.

Copied to clipboard

Feng Jiang, Weihao Liu, Xiaomin Chu, Peifeng Li, Qiaoming Zhu, Haizhou Li

M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database (2022.acl-long)

Challenge:	Compared with sentence-level topic structure, paragraph-level topics can grasp and understand the context of a document from a higher level.
Approach:	They propose a hierarchical paragraph-level topic structure representation with three layers to guide corpus construction.
Outcome:	The proposed method achieves the largest Chinese paragraph-level topic structure corpus, achieving high quality.

Copied to clipboard

Jinming Zhao, Tenggan Zhang, Jingwen Hu, Yuchen Liu, Qin Jin, Xinchao Wang, Haizhou Li

ChatCRS: Incorporating External Knowledge and Goal Guidance for LLM-based Conversational Recommender Systems (2025.findings-naacl)

Challenge:	Existing data resources to support multimodal affective analysis in dialogues are limited in scale and diversity.
Approach:	They propose a multimodal multi-scene multi-label Emotional Dialogue dataset, M3ED, which contains 990 dyadic emotional dialogues from 56 different TV series.
Outcome:	The proposed dataset contains 990 dyadic emotional dialogues from 56 different TV series, a total of 9,082 turns and 24,449 utterances.

Copied to clipboard

Chuang Li, Yang Deng, Hengchang Hu, Min-Yen Kan, Haizhou Li

Natural Language Interfaces

Challenge:	Experimental results show that ChatCRS improves language quality and informativeness by 17% and proactivity by 27%.
Approach:	They propose a framework to decompose the CRS task into several sub-tasks . they propose 'knowledge retrieval agent' and 'goal-planning agent'
Outcome:	The proposed framework improves language quality and informativeness by 17% and proactivity by 27% on two multi-goal CRS datasets.

How Well Do Text Embedding Models Understand Syntax? (2023.findings-emnlp)

Copied to clipboard

Yan Zhang, Zhaopeng Feng, Zhiyang Teng, Zuozhu Liu, Haizhou Li

Representation Learning

Challenge:	Existing text embedding models have not addressed syntactic understanding challenges, highlighting ineffectiveness and enhancing generalization ability.
Approach:	They propose to examine the ability of text embedding models to generalize across syntactic contexts.
Outcome:	The proposed models exhibit high similarity socres at this simple task.

CMB: A Comprehensive Medical Benchmark in Chinese (2024.naacl-long)

Copied to clipboard

Xidong Wang, Guiming Chen, Song Dingjie, Zhang Zhiyi, Zhihong Chen, Qingying Xiao, Junying Chen, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, Haizhou Li

Hanfu-Bench: A Multimodal Benchmark on Cross-Temporal Cultural Understanding and Transcreation (2025.emnlp-main)

Challenge:	Large Language Models (LLMs) provide a great breakthrough in medicine, says a new study . existing studies on LLMs leverage subjective evaluation, but evaluation in medicine is professional .
Approach:	They propose a localized medical benchmark in Chinese rooted in native Chinese . they propose to use traditional Chinese medicine to evaluate large-scale LLMs .
Outcome:	a new benchmark is developed to evaluate large-scale LLMs in china . the proposed model is rooted in the native Chinese linguistic and cultural framework .

Copied to clipboard

Li Zhou, Lutong Yu, Dongchu Xie, Shaohuan Cheng, Wenyan Li, Haizhou Li

NaturalSloth: Revisiting Denial-of-Service Attacks on Large Language Models (2026.acl-long)

Challenge:	Existing studies on cultural understanding with vision-language models primarily emphasize geographic diversity, often overlooking the critical temporal dimensions.
Approach:	They propose a multimodal vision-language model that examines temporal features and cultural image transcreation.
Outcome:	The novel model performs better than non-experts on visual cutural understanding but falls short to human experts on cultural image transcreation task.

Copied to clipboard

Yiming Chen, Zexin Li, Xianghu Yue, Robby T. Tan, Haizhou Li

Robustness in NLP

Challenge:	Longer generations consume more GPU time, increase latency, and reduce throughput in multi-tenant systems.
Approach:	They propose an adversarial dataset of natural instruction-based DoS prompts to scale the dataset while preserving malicious intent and increasing semantic diversity.
Outcome:	The proposed framework scales with a human-curated seed set of natural instruction-based DoS prompts while preserving malicious intent and increasing semantic diversity.

Analyzing and Evaluating Faithfulness in Dialogue Summarization (2022.emnlp-main)

Copied to clipboard

Bin Wang, Chen Zhang, Yan Zhang, Yiming Chen, Haizhou Li

AceGPT, Localizing Large Language Models in Arabic (2024.naacl-long)

Challenge:	Existing studies on faithfulness of text summarization have not been conducted on abstractive summarizing.
Approach:	They propose a method to evaluate faithfulness of dialogue summarization models by multi-choice questions.
Outcome:	The proposed method can facilitate the development of dialogue summarization systems.

Copied to clipboard

Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, Jinchao Xu

Uncovering the Potential of ChatGPT for Discourse Analysis in Dialogue: An Empirical Study (2024.lrec-main)

Challenge:	Significant concerns emerge when addressing cultural sensitivity and local values.
Approach:	They propose a localized Large Language Model (LLM) specifically for Arabic, a language imbued with unique cultural characteristics inadequately addressed by current mainstream models.
Outcome:	The proposed model sets the state-of-the-art standard for open Arabic LLMs across various benchmarks.

Copied to clipboard

Yaxin Fan, Feng Jiang, Peifeng Li, Haizhou Li