Papers by Yen-Chang Hsu
Adaptive Rank Selections for Low-Rank Approximation of Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Singular Value Decomposition (SVD) or its weighted variants has progressed in compressing language models. |
| Approach: | They propose a binary masking mechanism for optimizing the number of ranks in a differentiable framework. |
| Outcome: | The proposed algorithm achieves much better accuracy than previous SVD and its weighted variants. |
FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing (2025.naacl-long)
Copied to clipboard
James Seale Smith, Chi-Heng Lin, Shikhar Tuli, Haris Jeelani, Shangqian Gao, Yilin Shen, Hongxia Jin, Yen-Chang Hsu
| Challenge: | Empirical evaluations demonstrate substantial performance gains over existing methods . |
| Approach: | They propose a method to prune LLMs that selectively prunes model blocks based on an importance score and replaces them with a low-parameter replacement strategy. |
| Outcome: | The proposed method achieves state-of-the-art performance on 5/6 and 6/6 benchmarks with a compression rate of 30% and 40%. |
Numerical Optimizations for Weighted Low-rank Estimation on Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Singular value decomposition (SVD) is one of the most popular methods for estimating a target matrix with smaller matrices. |
| Approach: | They propose a method that approximates a target matrix with smaller matrices by two smaller . they also propose metric to predict when the SVD may introduce a significant performance drop. |
| Outcome: | The proposed method can perform better than current SOTA methods in compressing Transformer-based language models. |
DynaMo: Accelerating Language Model Inference with Dynamic Multi-Token Sampling (2024.naacl-long)
Copied to clipboard
| Challenge: | Rapid explosion in model sizes has resulted in high inference times . open-source LLMs are democratizing research in natural language processing . |
| Approach: | They propose a suite of multi-token prediction language models that reduce net inference times by leveraging traditional autoregressive weights. |
| Outcome: | The proposed model achieves same-quality generated text as baseline (Pythia-6.9B) with only 5.87% and 2.67% parameter and training time overheads. |
Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing continual learning approaches suffer from low accuracy and performance fluctuation when the distributions of old and new data are significantly different. |
| Approach: | They propose a hyperparameter-free continual learning model for text data that can stably produce high performance under various environments. |
| Outcome: | The proposed model outperforms the best state-of-the-art method by 20% in average accuracy and each component contributes effectively to overall performance. |
SLiM: Speculative Decoding with Hypothesis Reduction (2024.findings-naacl)
Copied to clipboard
| Challenge: | Speculative decoding has emerged as an alternative to autoregressive decoding for expediting inference in large language models (LLMs). prevailing assumptions focus solely on latency reduction, neglecting the computational expenses. |
| Approach: | They propose a speculative decoding enhancement to reduce the speculation set while validating more effective tokens. |
| Outcome: | The proposed method reduces the speculation set while validating more effective tokens. |
Dynamic Low-rank Estimation for Transformer-based Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | RankDyna is a matrix decomposition method that can be used to compress Transformer-based language models. |
| Approach: | They propose a matrix decomposition method that enables dynamic rank resource allocation . they say it can outperform current SOTA methods under various parameter budget levels . |
| Outcome: | The proposed method outperforms current SOTA methods under various budget levels . the proposed method is more efficient with higher compression rates . |
Enhancing the generalization for Intent Classification and Out-of-Domain Detection in SLU (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for intent classification are expensive to collect and train . evaluators have shown that the ability to detect out-of-domain utterances is limited . |
| Approach: | They propose to train a model with only IND data while supporting both intent classification and OOD detection. |
| Outcome: | The proposed model improves on existing models and strong baselines on four datasets. |