Papers by Juhua Liu
Zero-shot Sharpness-Aware Quantization for Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing zero-shot quantization methods are based on overfitting problem in adversarial learning process, leading to sub-optimal performance. |
| Approach: | They propose a zero-shot sharpness-aware quantization framework for the quantization of various PLMs by optimizing a minimax problem. |
| Outcome: | The proposed framework can achieve significant performance gains on discriminative and generative PLMs. |
Revisiting Knowledge Distillation for Autoregressive Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Autoregressive language models (LMs) are expensive and memory intensive, preventing the development of industrial applications. |
| Approach: | They propose an adaptive teaching approach to improve the KD of autoregressive language models by distilling knowledge into a small student model. |
| Outcome: | The proposed method can achieve consistent and significant performance gains across all model types and sizes. |
Revisiting Token Dropping Strategy in Efficient BERT Pretraining (2023.acl-long)
Copied to clipboard
| Challenge: | Token dropping is a recently-proposed strategy to speed up the pretraining of masked language models, such as BERT. |
| Approach: | They propose a semantic-consistent learning method to improve token dropping by skipping the computation of a subset of input tokens at several middle layers. |
| Outcome: | The proposed method achieves consistent and significant performance gains across all tasks and model sizes. |
KaFT: Knowledge-aware Fine-tuning for Boosting LLMs’ Domain-specific Question-Answering Performance (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent literature reveals that supervised fine-tuning (SFT) is suboptimal for domain-specific question-answering tasks. |
| Approach: | They propose a query diversification strategy for robust conflict detection and a knowledge-aware fine-tuning approach to effectively boost LLMs’ performance. |
| Outcome: | The proposed approach improves the model generalization and alleviates the hallucination. |
ROSE Doesn’t Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for aligning LLMs output with expected safety require substantial training efforts and expensive computational resources. |
| Approach: | They propose a method to directly boost the safety of existing instruction-tuned large language models without additional training. |
| Outcome: | The proposed method improves safety of instruction-tuned large language models without training and requires expensive computational resources. |
DYNTEXT: Semantic-Aware Dynamic Text Sanitization for Privacy-Preserving LLM Inference (2025.findings-acl)
Copied to clipboard
Juhua Zhang, Zhiliang Tian, Minghang Zhu, Yiping Song, Taishu Sheng, Siyi Yang, Qiunan Du, Xinwang Liu, Minlie Huang, Dongsheng Li
| Challenge: | Existing methods to protect privacy of sensitive data are differential privacy (DP) and DP is used to protect users from privacy leakage. |
| Approach: | They propose an LDP-based Dynamic Text sanitization for privacy-preserving LLM inference that dynamically constructs semantic-aware adjacency lists of sensitive tokens to sample non-sensitive tokens for perturbation. |
| Outcome: | The proposed model excels on three datasets. |
DPGA-TextSyn: Differentially Private Genetic Algorithm for Synthetic Text Generation (2025.findings-acl)
Copied to clipboard
Zhonghao Sun, Zhiliang Tian, Yiping Song, Yuyi Si, Juhua Zhang, Minlie Huang, Kai Lu, Zeyu Xiong, Xinwang Liu, Dongsheng Li
| Challenge: | Existing methods to fine-tune large language models pose privacy risks . researchers have synthesized data with strong generation capabilities closed-source LLMs to alleviate this problem . |
| Approach: | They propose to combine general LLMs with genetic algorithm to produce relevant and diverse synthetic text under differential privacy constraints. |
| Outcome: | The proposed method significantly improves the performance of the model in downstream tasks while ensuring privacy. |
Learning from Imperfect Data: Towards Efficient Knowledge Distillation of Autoregressive Language Models for Text-to-SQL (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text-to-SQL LLMs are computationally expensive and difficult to deploy in real-world applications. |
| Approach: | They propose to distill a larger teacher model into a smaller student model by using imperfect data to improve the KD. |
| Outcome: | The proposed method achieves the best tradeoff between performance and efficiency on 5 text-to-SQL benchmarks. |
Self-Evolution Learning for Discriminative Language Model Pretraining (2023.findings-acl)
Copied to clipboard
| Challenge: | Random masking does not consider the importance of the different words in the sentence meaning, e.g., entity-level masking requires expensive prior knowledge and generally does not use existing model weights. |
| Approach: | They propose a token masking and learning method that uses a random masking strategy to learn the under-explored tokens. |
| Outcome: | The proposed method improves linguistic knowledge learning and generalization on 10 tasks. |
Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pretrained language models suffer from poor generalization . however, they add a perturbation to each model parameter equally, which is sub-optimal . |
| Approach: | They propose a sharpness-aware minimization optimization procedure that introduces a Fisher mask to improve the efficiency of SAM. |
| Outcome: | The proposed method outperforms the vanilla sharpness-aware minimization method on GLUE and SuperGLUE benchmarks. |