Papers by Cheng-Han Chiang
Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course (2024.emnlp-main)
Copied to clipboard
| Challenge: | Using large language models (LLMs) for automatic evaluation has become an important evaluation method in NLP research. |
| Approach: | They use large language models (LLMs) for automatic evaluation to evaluate a sample . they propose several recommendations for integrating LLMs into future classroom evaluations . |
| Outcome: | The proposed model is able to output high scores without meeting the evaluation instructions, the authors note . their model is not able for students to manipulate the model to output specific strings, they say . |
Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing factuality metrics cannot evaluate paragraphs with ambiguous entities, authors show . |
| Approach: | They propose a new metric to evaluate the factuality of long-form generations from large language models. |
| Outcome: | The proposed metric can assess the factuality of people biographies with entity ambiguity better than FActScore. |
Pretrained Language Model Embryology: The Birth of ALBERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining of pretrained models (LMs) has been extensively studied, but what happened during pretraining is rarely studied. |
| Approach: | They propose to use a totipotent language model to study pretraining behavior . they find that linguistic knowledge and world knowledge do not generally improve as pretraining proceeds, nor do downstream tasks’ performance. |
| Outcome: | The model learns to reconstruct and predict tokens of different parts of speech (POS) in different learning speeds during pretraining. |
Can Large Language Models Be an Alternative to Human Evaluations? (2023.acl-long)
Copied to clipboard
| Challenge: | Human evaluation is indispensable for assessing the quality of texts generated by machine learning models or written by humans. |
| Approach: | They propose to use large language models to evaluate unseen texts using the same instructions and samples . they also use LLMs to generate responses to questions that are used to conduct human evaluation . |
| Outcome: | The proposed model can be used to evaluate texts in open-ended story generation and adversarial attacks. |
Are Synonym Substitution Attacks Really Synonym Substitution Attacks? (2023.findings-acl)
Copied to clipboard
| Challenge: | In synonym substitution attacks, an adversarial sample is constructed by substituting words in the original sentence with their synonyms. |
| Approach: | They examine how synonym substitution attacks replace words in the original sentence and show that there are still unresolved obstacles that make current SSAs generate invalid adversarial samples. |
| Outcome: | The proposed methods generate large fractions of invalid substitution words that are ungrammatical or do not preserve the original sentence’s semantics. |
Audio-Aware Large Language Models as Judges for Speaking Styles (2025.findings-emnlp)
Copied to clipboard
Cheng-Han Chiang, Xiaofei Wang, Chung-Ching Lin, Kevin Lin, Linjie Li, Radu Kopetz, Yao Qian, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, Lijuan Wang
| Challenge: | Audio-aware large language models (ALLMs) can understand textual and non-textual information in the audio input. |
| Approach: | They use audio-aware large language models (ALLMs) to evaluate the speaking styles of SLMs on two tasks: voice style instruction following and role-playing. |
| Outcome: | The proposed models can understand the textual and non-textual information in the audio input and can be used as a judge to assess the speaking styles of SLMs. |
Style Amnesia: Investigating Speaking Style Degradation and Mitigation in Multi-Turn Spoken Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Unlike textonly large language models (LLMs), SLMs integrate audio encoders and vocoders to support end-to-end speech understanding and generation. |
| Approach: | They evaluate three proprietary and two open-source SLMs and show that none of them can maintain a consistent speaking style when instructed to do so. |
| Outcome: | The proposed models cannot maintain a consistent speaking style after several turns of interaction, but can recall the style instruction when prompted in later turns, but fail to express it. |
Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations (2024.acl-long)
Copied to clipboard
| Challenge: | In spoken dialogue, even if two current turns are the same sentence, their responses might differ when they are spoken in different styles. |
| Approach: | They propose a language-to-speech dataset that can model linguistic content and speaking styles. |
| Outcome: | The proposed framework outperforms text-only baselines and prior speech LLMs methods. |
TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judge (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning LLMs use cross-entropy (CE) loss . existing methods neglect the numeric nature of score prediction . |
| Approach: | They propose a method that fine-tunes large language models (LLMs) for automated text evaluation, assigning a score to the input based on scoring rubrics. |
| Outcome: | The proposed model outperforms existing methods in four LLM-as-a-judge datasets and two LLMs. |
A Closer Look into Using Large Language Models for Automatic Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies explore the use of large language models to evaluate text quality, but they differ in some details of the evaluation process. |
| Approach: | They propose to use large language models to evaluate text quality by giving LLMs instructions to evaluate samples by giving them a rating. |
| Outcome: | The auto Chain-of-Thought (CoT) used in G-Eval does not always make it more aligned with human ratings. |
Shanks: Simultaneous Hearing and Thinking for Spoken Language Models (2026.acl-long)
Copied to clipboard
Cheng-Han Chiang, Xiaofei Wang, Linjie Li, Chung-Ching Lin, Kevin Lin, Shujie Liu, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, Lijuan Wang
| Challenge: | Existing large language models and spoken language models (SLMs) begin thinking and taking actions only after the user has finished their turn. |
| Approach: | They propose a general inference framework that enables SLMs to generate unspoken chain-of-thought reasoning while listening to user input. |
| Outcome: | The proposed framework enhances real-time user–SLM interaction in two scenarios. |
Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | Pre-trained language models are language models that are pre-taught on large-scaled corpora in a self-supervised fashion. |
| Approach: | This tutorial provides a broad and comprehensive introduction to pre-trained language models . it focuses on emerging methods that enable PLMs to perform diverse downstream tasks . |
| Outcome: | This tutorial focuses on the benefits of pre-trained language models and how to use them in NLP tasks. |
Over-Reasoning and Redundant Calculation of Large Language Models (2024.eacl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) can solve problems step-by-step, but it is unclear whether they know when to use CoT and whether they are always necessary. |
| Approach: | They propose to use LLMs to generate redundant calculations and reasoning on a manually constructed math QA dataset, GSM8K-Zero. |
| Outcome: | The proposed model generates redundant calculations and reasoning on a manually constructed math QA dataset, but it is unclear whether it is necessary to use CoT reasoning. |