Papers by Tianyu Pang
HTMuon: Improving Muon via Heavy-Tailed Spectral Correction (2026.findings-acl)
Copied to clipboard
| Challenge: | Muon’s orthogonalized update rule suppresses the emergence of heavy-tailed weight spectra and over-emphasizes the training along noise-dominated directions. |
| Approach: | They propose to preserve Muon's ability to capture parameter interdependencies while producing heavier-tailed updates and inducing heavier-tail weight spectra. |
| Outcome: | The proposed algorithm suppresses the emergence of heavy-tailed weight spectra and over-emphasizes training along noise-dominated directions. |
Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning (2024.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that fine-tuning of large language models for specific tasks can be challenging . distribution shift during fine-timing can lead to performance degradation in general task capabilities . |
| Approach: | They propose a new approach that bridges the distribution gap between task datasets and LLMs by guiding fine-tuning with a distilled dataset generated by the model itself. |
| Outcome: | The proposed approach achieves comparable or superior performance on downstream tasks compared to the vanilla approach. |
LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can process extremely long contexts, requiring efficient inference over extended inputs. |
| Approach: | They propose a model that uses a constant-sized key-value cache to train long-context models. |
| Outcome: | Experimental results show that LongSpec achieves 3.26x speedup over strong Flash Attention baselines and 2.34x wall clock time on four math reasoning tasks. |
Model Balancing Helps Low-data Training and Fine-tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in foundation models have emphasized the need to align pre-trained models with specialized domains using small, curated datasets. |
| Approach: | They propose a layer-wise learning rate scheduler that balances training quality across layers . they adapt it to a curated dataset to achieve alignment with specialized domains . |
| Outcome: | The proposed model shows that it can be used to balance training quality across layers and improve low-data training and fine-tuning for both NLP and SciML tasks. |
Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasets (2026.acl-long)
Copied to clipboard
| Challenge: | Existing mitigation strategies focus on reactively addressing jailbreak incidents after safety guardrails have been compromised. |
| Approach: | They investigate the degradation of safety guardrails through the lens of representation similarity between upstream alignment datasets and downstream fine-tuning tasks. |
| Outcome: | The proposed model reduces harmfulness score by 10.33% when compared to baseline models. |